patch agent hit rate limit mid-fetch of vendor cve metadata and failed

TL;DR

CVE metadata is reference data URIs it changes slowly, is identical for every wave, and is heavily rate limited when fetched without an API key (the NVD API is the classic case). Fetch it like reference data URIs use an API key, cache responses keyed by CVE ID with a long TTL, batch and pace requests under the published limit, and decouple planning from fetching so a throttle degrades the wave (plan with cached data and note staleness) instead of killing it.

patch agent hit rate limit mid-fetch of vendor cve metadata and failed

Use this when

• Planning logs show 403 or 429 responses from the CVE API after a burst of lookups.
• The same CVEs are fetched fresh every wave.
• The run works for small patches and fails on Patch Tuesday scale.

Not for this skill when

• The CVE source changed its schema or moved endpoints. That is an API change, not a rate problem.
• Planning fails because the patch data itself is missing. CVE enrichment is secondary; do not let it block core data.

Steps

1. Confirm the API rate terms (with and without a key) and instrument how many requests a planning run makes and for which CVEs. Verify: request count and unique CVE count are both known
2. Obtain and configure an API key through the vault if the vendor offers one, and set the client rate below the keyed limit. Verify: the client identifies itself and the higher limit applies
3. Add a persistent cache keyed by CVE ID storing the fields planning uses, with a TTL of days and a stale-while-revalidate behavior. Verify: a second planning run shows near-total cache hits
4. Make enrichment best-effort in the planner: missing or stale CVE data annotates the plan with a staleness note rather than failing it. Verify: a run with the API fully blocked still produces a plan using cache, clearly labeled
5. Pre-warm the cache on a schedule (for example nightly for newly published CVEs in your KB set) so wave time is not fetch time. Verify: wave planning performs zero cold fetches in the normal case

Variant phrasings

nvd api rate limit cve metadata cache agent planning

The main phrasing.

api key higher rate limit nvd requests per window

The key phrasing.

decouple enrichment from planning stale cache degrade

The resilience phrasing.

Compatibility: Compatibility: NVD API 2.0 and vendor advisory APIs; cache can be sqlite or a JSON store beside the agent.

Why it happens

Agents treat APIs as infinite and instant; public vulnerability APIs are neither, and they are shared with every other agent on the internet doing the same Patch Tuesday lookups. The deeper design point is dependency direction: enrichment data should never be a hard dependency of the core task. Caching plus graceful degradation converts a fragile external dependency into a background process that can fail quietly without taking the wave with it.

Edge cases / pitfalls

• Respect the vendor terms and published limits even when unenforced; fleets get IPs banned, and a banned source helps nobody.
• CVE records do get revised (scoring and references change); a TTL of days, not forever, balances load against accuracy.
• If the organization runs a vulnerability scanner, its feed is a better internal source than the public API and has no public rate limit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_F6Vl2Rs24mxQIFTDdBqcag