fraud agent failed mid-review queue on sift score api rate limit

TL;DR

The agent asked for scores faster than the quota allows. Add a client-side rate limiter with jittered backoff on 429, cache scores per user for the run, and checkpoint the queue position after each item so a crash resumes instead of restarting.

fraud agent failed mid-review queue on sift score api rate limit

Use this when

• The agent 429s partway through a review batch.
• Restarting the run re-scores from zero and hits the limit again.
• The failure clusters at the start of the run, when the burst is hottest.

Not for this skill when

• Sift itself is down. That is a provider outage, not a client pacing problem.
• The API key is invalid. That fails on the first call, not mid-queue.
• Scores return but look wrong. That is a model or integration issue.

Steps

1. Checkpoint first: persist the last fully processed item id after every item, in durable storage. Verify: killing the agent mid-run loses at most one item.
2. Add a token-bucket limiter on the Sift client set below your quota, with exponential backoff and jitter on 429. Verify: a burst of requests smooths out instead of erroring.
3. Cache score responses per user id with a short TTL for the duration of the run. Verify: repeat lookups for the same user never hit the API twice.
4. On resume, read the checkpoint and skip already-processed items. Verify: a restarted run continues from the checkpoint, not from zero.
5. Alert when backoff engages repeatedly. Verify: sustained throttling pages someone instead of silently slowing the queue.

Variant phrasings

sift 429 killed review agent

Shorter phrasing of the same failure.

fraud agent rate limited by sift

Symptom-level search.

resume review queue after sift rate limit

Recovery-side wording.

Compatibility: Sift Score API. Client-side throttling; the quota itself is set by your Sift plan.

Why it happens

Review agents fire one score request per queue item as fast as they can loop, which is a burst pattern API quotas are designed to punish. Without a limiter the agent burns the quota in seconds, and without a checkpoint every crash means re-scoring the whole queue, which burns it again.

Edge cases / pitfalls

• Do not raise concurrency to catch up after a pause. That re-triggers the limit.
• Fail-open versus fail-closed on score failure is a risk-team decision. Make it explicit before the next incident.
• Cache TTLs must be short. A stale score on a fast-moving fraudster is worse than a slow one.
• Separate your agent's quota from interactive traffic if you can, so a runaway run does not starve the checkout path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_27yaouxd002-9fUlQsiqVw