security agent's CVE feed fetch hit the NVD API rate limit: backoff strategy

security agent's CVE feed fetch hit the NVD API rate limit: backoff strategy

TL;DR

The agent hammered the NVD API without a key and got rate-limited. Request an NVD API key (it raises the limit substantially), pace the fetches, honor retry-after, and cache aggressively. Unkeyed, unpaced NVD polling always ends in 429.

NVD API rate limit hit during CVE feed fetch (429/403 on api.nvd.nist.gov)

Steps

1. Confirm the throttle: check the response code and any rate-limit headers. Expected: The limit state is visible.

2. Get an NVD API key and send it on every request; the keyed limit is far higher. Expected: The ceiling rises.

3. Pace the fetches: delay between pages, and never poll more often than the feed updates. Expected: Requests stay under the limit.

4. Honor retry-after on 429 with backoff and jitter. Expected: Throttles pause instead of failing.

5. Cache the feed locally and serve the agent from the cache; fetch incrementally. Expected: The API is hit rarely.

When to use

• NVD API calls get rate-limited
• CVE syncs run unkeyed
• the feed is polled aggressively

When not to use

• the limit is on a different API (check the host)
• keyed requests still 429 (check for a loop)
• the data is available from a mirror (use it)

Tool compatibility

• NVD API 2.0
• CVE feed sync

Variant phrasings

• "NVD API rate limit backoff"
• "CVE feed fetch throttled"
• "nvd api 429 api key"

Why it happens

The NVD API rate-limits aggressively, especially unkeyed requests, and CVE feeds are large enough that a naive full pull exceeds it. The key exists precisely for automation; without it the agent is a guest overstaying. Caching turns the feed from a hammer into a background sync.

Edge cases

• the API key must be sent on every request, not just some
• incremental sync needs the last-modified watermark persisted
• NVD has maintenance windows; distinguish those from rate limits

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEO62aFknZg4wy5DU2uMhA