fraud agent timed out during riskified batch review under load

TL;DR

Big batches plus high concurrency equals timeouts. Split the batch into small chunks, cap parallel workers, give each order its own timeout, and checkpoint per chunk so a stall only costs one chunk.

fraud agent timed out during riskified batch review under load

Use this when

• Large review batches time out partway under load.
• The same batch size works off-peak and fails at peak.
• One slow order stalls everything behind it.

Not for this skill when

• Riskified decisions look wrong. That is decision logic, not pacing.
• Webhooks are delayed. That is a delivery problem.
• Single-order reviews time out. That is a per-call problem.

Steps

1. Chunk the batch into small groups, for example 25 orders per chunk. Verify: a single chunk completes comfortably inside the timeout.
2. Cap worker concurrency below the point where timeouts start. Verify: raising concurrency no longer raises the timeout rate.
3. Set a per-order timeout inside the chunk loop so one slow order cannot stall the chunk. Verify: slow orders are flagged, not fatal.
4. Checkpoint after each chunk with the last completed order id. Verify: a restart skips finished chunks.
5. Process high-value orders first so the most important reviews land even if the run degrades. Verify: the queue order reflects value, not arrival time.

Variant phrasings

riskified batch timeout agent

Same failure, common phrasing.

bulk review timed out riskified

Symptom-level search.

agent slow on riskified batch

Performance-side wording.

Compatibility: Riskified order review API consumed by an agent batch loop.

Why it happens

Batch loops multiply latency: one slow order blocks every order behind it, and under load the API itself slows down, so the batch that finished in two minutes at 9am times out at 2pm. Without chunking and checkpoints, one timeout discards the whole batch's progress and the retry starts from zero. Chunking also bounds the blast radius: a provider slowdown that would have killed a 500-order batch now costs one 25-order chunk.

Edge cases / pitfalls

• Reordering by value can starve low-value orders. Set a maximum wait before they get processed anyway.
• Per-order timeouts that are too tight create false unreviewable flags. Tune from p99 latency.
• Keep the human review SLA in mind. A checkpointed slow run still has to finish before the SLA.
• Do not grow chunk size back up after one good run. Load varies by hour.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XqTjO2o7sYaYMxVLf6xI9Q