fraud agent stuck retrying a failed webhook delivery forever
Ends infinite webhook retry loops with bounded exponential backoff, dead-letter queues, and circuit breakers. Use when the agent retries a failed delivery without limit. Not for one-off delivery failures that succeed on retry.
TL;DR
A webhook destination is down and the agent retries on a tight loop with no exit. Fix: bound the retries with exponential backoff plus jitter, classify failures so permanent ones stop immediately, and send survivors to a dead-letter queue with their failure history. A retry loop without a ceiling is a bug, not resilience.
fraud agent stuck retrying a failed webhook delivery foreverUse this when
- One delivery id accumulates attempts with no bound in sight.
- A dead destination slows the whole queue because workers are stuck on it.
- Retry logs show constant intervals hammering an endpoint.
Not for this skill when
- A delivery fails once or twice and then succeeds. That is a healthy retry.
- The endpoint is up but slow. That is a timeout tuning problem.
Steps
- Confirm the loop: find the delivery id being retried and count its attempts. Verify: attempts exceed any sane bound, like ten.
- Set a max attempt count of five to eight with exponential backoff from about one second, plus jitter. Verify: retry intervals grow and vary between attempts.
- Classify failures: client errors go straight to the dead-letter queue with no retries, except 408 and 429 which are retryable. Verify: a 400 is never retried.
- After the final attempt, write the payload to the dead-letter queue with the full failure history and alert once. Verify: the payload is inspectable and replayable by hand.
- Add a circuit breaker: when one destination fails repeatedly, pause all deliveries to it for a window. Verify: a single bad endpoint stops poisoning the whole queue.
Variant phrasings
webhook retry loop never stops
Common phrasing.
infinite webhook redelivery
Delivery-focused search.
agent retries failed webhook forever
Agent-focused phrasing.
Compatibility: any webhook sender; the pattern is identical whether the queue is SQS, RabbitMQ, or a database table.
Why it happens
The retry code was written for the happy path, where the endpoint blipped and one more try fixes it. Nobody defined what happens when the endpoint is really gone, so "keep trying" quietly became the entire program.
Edge cases / pitfalls
- Honor the Retry-After header on 429s; do not lump rate limits in with other client errors.
- Poison payloads that crash the sender need a catch around the send, a DLQ write, and the queue keeps moving.
- Replaying the DLQ later needs the same idempotency keys, or replays double-apply.
Provenance
Resolved from the public thread: https://vectle.com/posts/pstLoSN8D5PjqWbtyMZno8YA