the EPSS API throttled the agent's enrichment step - 10,000 CVEs queued behind a 100-per-minute limit

TL;DR

Download the EPSS scores in bulk instead of querying the API per CVE. FIRST publishes a daily CSV with every score; load it into a local store once a day and serve enrichment from there. The API is for one-off lookups, not for 10,000-CVE backlogs.

the EPSS API throttled the agent's enrichment step - 10,000 CVEs queued behind a 100-per-minute limit

Steps

1. Do the math on the current approach. Divide your CVE count by the per-minute API limit to get the floor on enrichment time.

   • Check: 10,000 CVEs at 100 per minute is 100 minutes minimum, before retries or failures.
   • Expected: the number confirms the API path cannot fit your window.

2. Switch to the bulk download. Fetch the daily EPSS CSV file published by FIRST and load it into a local table keyed by CVE ID.

   • Run: download the current epss_scores-YYYY-MM-DD.csv.gz from the EPSS data site, decompress, and import the cve, epss, and percentile columns.
   • Expected: one download covers every CVE; import takes minutes.

3. Point the enrichment step at the local table. The agent looks up each CVE's score and percentile locally instead of calling the API.

   • Expected: enrichment of 10,000 CVEs runs in seconds to minutes.

4. Refresh the table daily with the newest CSV. Scores move slowly; a once-a-day refresh is the documented cadence and keeps data fresh enough for triage.

   • Expected: scores never more than about a day old.

5. Keep the API only for just-published CVEs that have not landed in the CSV yet. A tiny allowlist of recent IDs goes to the API; everything else hits the table.

   • Expected: a handful of API calls per run, well under the limit.

6. Record the score date on every enriched finding so downstream consumers know the data's age.

   • Expected: each finding carries its EPSS as-of date.

Use this when

• EPSS enrichment of a large backlog is throttled or too slow via the API.
• The enrichment step's runtime is dominated by per-CVE EPSS lookups.
• You enrich on a schedule (nightly, hourly) rather than interactively.
• API rate limits, not data availability, are the bottleneck.

Not for this skill when

• You look up a handful of CVEs interactively; the API is the right tool there.
• You need EPSS history or model metadata; the bulk CSV carries current scores only.
• The throttle is on a different API in the chain (NVD, OSV); fix that one instead.

Variant phrasings

• EPSS API rate limit during CVE enrichment
• how to bulk download EPSS scores
• enriching thousands of CVEs with EPSS too slow
• FIRST EPSS API throttling automation

Why it happens

The EPSS API is rate-limited per minute and designed for ad-hoc lookups, while FIRST also publishes the full score set as a daily CSV for exactly the bulk use case. An agent that treats the API as a database and queries it 10,000 times is using the wrong interface; the queue behind the per-minute limit is the API working as designed against a misuse pattern.

Edge cases

• Scores for brand-new CVEs may lag the CSV by a day; the API fallback covers the gap.
• The CSV schema has changed before (header formats); validate the header on import and alert on unexpected columns.
• Percentile and probability answer different questions; make sure the agent's priority math uses the right one (probability for likelihood, percentile for ranking).

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bC3rmdR-UuWwN4n84ZkE9Q