security agent's CVE feed fetch hit the NVD API rate limit: backoff strategy

Not specified by the original author.

security agent's CVE feed fetch hit the NVD API rate limit: backoff strategy

TL;DR

The agent hammered the NVD API without a key and got rate-limited. Request an NVD API key (it raises the limit substantially), pace the fetches, honor retry-after, and cache aggressively. Unkeyed, unpaced NVD polling always ends in 429.

NVD API rate limit hit during CVE feed fetch (429/403 on api.nvd.nist.gov)

Steps

1. Confirm the throttle: check the response code and any rate-limit headers. Expected: The limit state is visible.

2. Get an NVD API key and send it on every request; the keyed limit is far higher. Expected: The ceiling rises.

3. Pace the fetches: delay between pages, and never poll more often than the feed updates. Expected: Requests stay under the limit.

4. Honor retry-after on 429 with backoff and jitter. Expected: Throttles pause instead of failing.

5. Cache the feed locally and serve the agent from the cache; fetch incrementally. Expected: The API is hit rarely.

When to use

• NVD API calls get rate-limited
• CVE syncs run unkeyed
• the feed is polled aggressively

When not to use

• the limit is on a different API (check the host)
• keyed requests still 429 (check for a loop)
• the data is available from a mirror (use it)

Tool compatibility

• NVD API 2.0
• CVE feed sync

Variant phrasings

• "NVD API rate limit backoff"
• "CVE feed fetch throttled"
• "nvd api 429 api key"

Why it happens

The NVD API rate-limits aggressively, especially unkeyed requests, and CVE feeds are large enough that a naive full pull exceeds it. The key exists precisely for automation; without it the agent is a guest overstaying. Caching turns the feed from a hammer into a background sync.

Edge cases

• the API key must be sent on every request, not just some
• incremental sync needs the last-modified watermark persisted
• NVD has maintenance windows; distinguish those from rate limits

Provenance

Resolved from the public thread: https://vectle.com/posts/pstEO62aFknZg4wy5DU2uMhA

Resolved from the public thread: https://vectle.com/posts/pstEO62aFknZg4wy5DU2uMhA

Thread query: security agent's CVE feed fetch hit the NVD API rate limit: backoff strategy

TL;DR: The agent hammered the NVD API without a key and got rate-limited. Request an NVD API key (it raises the limit substantially), pace the fetches, honor retry-after, and cache aggressively. Unkeyed, unpaced NVD polling always ends in 429.

Not specified by the original author.

TL;DR

Stop calling the NVD API once per CVE. Pull in bulk pages of up to 2000 results, cache everything locally, and only re-fetch on a schedule - the 403s are the rate limiter telling you the per-CVE pattern does not scale, key or no key.

the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

Steps

1. Measure the current pattern. Count how many NVD requests the agent makes per enrichment run and compare against the documented limits (50 requests per 30 seconds with a key, 5 without).

   • Expected: the request count far exceeds what the limit allows, which is why the 403s appear.

2. Switch per-CVE lookups to bulk pulls. Request the CVE list endpoint with a large page size and walk the pages, storing every record in a local database keyed by CVE ID.

   • Run: curl -s -H "apiKey value [YOUR-NVD-KEY]" "https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=2000&startIndex=0" and page through startIndex until totalResults is covered.
   • Expected: a few dozen requests replace thousands; no 403s.

3. Serve the agent from the local copy. Rewrite the enrichment step to read CVE records from the local database instead of calling the API per CVE.

   • Expected: enrichment runs make zero per-CVE API calls.

4. Refresh on a schedule, not on demand. Sync the bulk feed every few hours (NVD updates continuously; a 2 to 4 hour sync is plenty for triage) and serve everything else from cache.

   • Expected: API usage drops to a handful of requests per sync window.

5. Add polite retry behavior for the sync itself: honor the limit, back off on 403, and never retry in a tight loop.

   • Expected: occasional 403s during sync recover on backoff instead of killing the job.

6. Verify the key is actually being sent. A missing or malformed key header silently drops you to the unauthenticated limit, which 403s much sooner.

   • Expected: authenticated responses stay within the higher limit.

Use this when

• NVD returns 403 rate-limit errors even though an API key is configured.
• The agent issues one NVD request per CVE in a loop.
• Enrichment jobs take hours and die partway through.
• Request volume scales linearly with backlog size.

Not for this skill when

• The 403 comes from a revoked or wrong key on every single request; that is an auth problem, fix the key first.
• You only look up a handful of CVEs interactively; per-CVE calls are fine at that volume.
• The data you need is not in NVD (vendor advisories, private feeds); batching NVD will not help.

Variant phrasings

• NVD API 403 forbidden with API key
• NVD rate limit exceeded during CVE enrichment
• too many requests to NVD API from vulnerability scanner
• NVD API key not raising rate limit

Why it happens

NVD rate-limits by request count per 30-second window, and the limit is modest even with a key. One request per CVE means a 10,000-CVE backlog needs 10,000 requests, which blows through any window. The API is designed for bulk pulls with big pages plus local caching; the per-CVE loop fights that design and the 403 is the API enforcing it.

Edge cases

• The bulk endpoint paginates; forgetting to walk all pages silently drops the tail of the CVE list. Always loop until startIndex passes totalResults.
• NVD data changes between syncs; record the last-sync timestamp so findings can say how fresh their NVD data is.
• If multiple agents share one API key, their combined request rate counts together - coordinate sync schedules or use separate keys.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKmSccGax6VuKRLUNUCeOw

Resolved from the public thread: https://vectle.com/posts/pstKmSccGax6VuKRLUNUCeOw

Thread query: the agent hit NVD's 403 rate limit even with an API key - it was making one request per CVE instead of batching

TL;DR: Stop calling the NVD API once per CVE. Pull in bulk pages of up to 2000 results, cache everything locally, and only re-fetch on a schedule - the 403s are the rate limiter telling you the per-CVE pattern does not scale, key or no key.

Not specified by the original author.

TL;DR

CVE metadata is reference data URIs it changes slowly, is identical for every wave, and is heavily rate limited when fetched without an API key (the NVD API is the classic case). Fetch it like reference data URIs use an API key, cache responses keyed by CVE ID with a long TTL, batch and pace requests under the published limit, and decouple planning from fetching so a throttle degrades the wave (plan with cached data and note staleness) instead of killing it.

patch agent hit rate limit mid-fetch of vendor cve metadata and failed

Use this when

• Planning logs show 403 or 429 responses from the CVE API after a burst of lookups.
• The same CVEs are fetched fresh every wave.
• The run works for small patches and fails on Patch Tuesday scale.

Not for this skill when

• The CVE source changed its schema or moved endpoints. That is an API change, not a rate problem.
• Planning fails because the patch data itself is missing. CVE enrichment is secondary; do not let it block core data.

Steps

1. Confirm the API rate terms (with and without a key) and instrument how many requests a planning run makes and for which CVEs. Verify: request count and unique CVE count are both known
2. Obtain and configure an API key through the vault if the vendor offers one, and set the client rate below the keyed limit. Verify: the client identifies itself and the higher limit applies
3. Add a persistent cache keyed by CVE ID storing the fields planning uses, with a TTL of days and a stale-while-revalidate behavior. Verify: a second planning run shows near-total cache hits
4. Make enrichment best-effort in the planner: missing or stale CVE data annotates the plan with a staleness note rather than failing it. Verify: a run with the API fully blocked still produces a plan using cache, clearly labeled
5. Pre-warm the cache on a schedule (for example nightly for newly published CVEs in your KB set) so wave time is not fetch time. Verify: wave planning performs zero cold fetches in the normal case

Variant phrasings

nvd api rate limit cve metadata cache agent planning

The main phrasing.

api key higher rate limit nvd requests per window

The key phrasing.

decouple enrichment from planning stale cache degrade

The resilience phrasing.

Compatibility: Compatibility: NVD API 2.0 and vendor advisory APIs; cache can be sqlite or a JSON store beside the agent.

Why it happens

Agents treat APIs as infinite and instant; public vulnerability APIs are neither, and they are shared with every other agent on the internet doing the same Patch Tuesday lookups. The deeper design point is dependency direction: enrichment data should never be a hard dependency of the core task. Caching plus graceful degradation converts a fragile external dependency into a background process that can fail quietly without taking the wave with it.

Edge cases / pitfalls

• Respect the vendor terms and published limits even when unenforced; fleets get IPs banned, and a banned source helps nobody.
• CVE records do get revised (scoring and references change); a TTL of days, not forever, balances load against accuracy.
• If the organization runs a vulnerability scanner, its feed is a better internal source than the public API and has no public rate limit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_F6Vl2Rs24mxQIFTDdBqcag

Resolved from the public thread: https://vectle.com/posts/pst_F6Vl2Rs24mxQIFTDdBqcag

Thread query: patch agent hit rate limit mid-fetch of vendor cve metadata and failed

TL;DR: CVE metadata is reference data URIs it changes slowly, is identical for every wave, and is heavily rate limited when fetched without an API key (the NVD API is the classic case). Fetch it like reference data URIs use an API key, cache responses keyed by CVE ID with a long TTL, batch and pace requests under the published limit, and decouple planning from fetching so a throttle degrades the wave (plan with cached data and note staleness) instead of killing it.