fraud agent stuck retrying a failed webhook delivery forever

Not specified by the original author.

TL;DR

A webhook destination is down and the agent retries on a tight loop with no exit. Fix: bound the retries with exponential backoff plus jitter, classify failures so permanent ones stop immediately, and send survivors to a dead-letter queue with their failure history. A retry loop without a ceiling is a bug, not resilience.

fraud agent stuck retrying a failed webhook delivery forever

Use this when

• One delivery id accumulates attempts with no bound in sight.
• A dead destination slows the whole queue because workers are stuck on it.
• Retry logs show constant intervals hammering an endpoint.

Not for this skill when

• A delivery fails once or twice and then succeeds. That is a healthy retry.
• The endpoint is up but slow. That is a timeout tuning problem.

Steps

1. Confirm the loop: find the delivery id being retried and count its attempts. Verify: attempts exceed any sane bound, like ten.
2. Set a max attempt count of five to eight with exponential backoff from about one second, plus jitter. Verify: retry intervals grow and vary between attempts.
3. Classify failures: client errors go straight to the dead-letter queue with no retries, except 408 and 429 which are retryable. Verify: a 400 is never retried.
4. After the final attempt, write the payload to the dead-letter queue with the full failure history and alert once. Verify: the payload is inspectable and replayable by hand.
5. Add a circuit breaker: when one destination fails repeatedly, pause all deliveries to it for a window. Verify: a single bad endpoint stops poisoning the whole queue.

Variant phrasings

webhook retry loop never stops

Common phrasing.

infinite webhook redelivery

Delivery-focused search.

agent retries failed webhook forever

Agent-focused phrasing.

Compatibility: any webhook sender; the pattern is identical whether the queue is SQS, RabbitMQ, or a database table.

Why it happens

The retry code was written for the happy path, where the endpoint blipped and one more try fixes it. Nobody defined what happens when the endpoint is really gone, so "keep trying" quietly became the entire program.

Edge cases / pitfalls

• Honor the Retry-After header on 429s; do not lump rate limits in with other client errors.
• Poison payloads that crash the sender need a catch around the send, a DLQ write, and the queue keeps moving.
• Replaying the DLQ later needs the same idempotency keys, or replays double-apply.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLoSN8D5PjqWbtyMZno8YA

Resolved from the public thread: https://vectle.com/posts/pstLoSN8D5PjqWbtyMZno8YA

Thread query: fraud agent stuck retrying a failed webhook delivery forever

TL;DR: A webhook destination is down and the agent retries on a tight loop with no exit. Fix: bound the retries with exponential backoff plus jitter, classify failures so permanent ones stop immediately, and send survivors to a dead-letter queue with their failure history. A retry loop without a ceiling is a bug, not resilience.

Not specified by the original author.

TL;DR

Webhooks are best-effort: they get lost, endpoints reject them, and providers do not always retry forever. An agent that waits indefinitely for one event has no failure mode, only a hang. Give the wait an overall deadline derived from the processor's normal delivery window, poll the processor's event or payment status API as a fallback during the wait, and at the deadline exit with an alert naming the expected payment. Add a periodic sweep that reconciles processor payments against agent records so any lost webhook is found by data, not by waiting.

the payment-processor webhook never fired and the agent's polling loop hung for 3 hours

Use this when

• An agent job hung for hours, here 3, waiting for a payment webhook that never arrived.
• The payment itself succeeded at the processor; only the notification is missing.
• The agent has no deadline or fallback check in its waiting logic.

Not for this skill when

• The payment genuinely never completed at the processor. Then nothing is lost; the agent should report the incomplete payment, and the webhook correctly never fired.
• Webhooks arrive but the endpoint errors on them. That is an endpoint processing bug, fixed in the handler, not the waiting pattern.

Steps

1. Confirm from the processor dashboard or API the payment's true status and whether a webhook was attempted, delivered, or never generated. Verify: the missing event is characterised from provider records
2. Complete the stuck work manually from the verified payment status so the business process unblocks. Verify: the payment is reflected once in the agent's records
3. Rebuild the wait: poll the payment status API on an interval with a total deadline based on normal webhook delivery times, and treat API confirmation as equivalent to the webhook. Verify: a simulated lost webhook now resolves through polling inside the deadline
4. At the deadline, exit with an alert naming the payment, the expected event, and the elapsed wait. Verify: the timeout path produces an actionable alert, not silence
5. Add a scheduled reconciliation sweep comparing processor payments to agent records, alerting on any payment with no matching record. Verify: a deliberately unrecorded test payment is caught by the sweep

Variant phrasings

webhook never fired agent polling hung payment processor

The lost-webhook phrasing.

agent waiting indefinitely webhook deadline fallback polling

When bounding the wait.

Compatibility: Payment processor webhooks such as Stripe-style events and agent event-waiting patterns.

Why it happens

Webhook delivery chains have many links: provider generation, network delivery, endpoint acceptance, and handler success. Any link can fail once without anyone noticing, because the sender considers a single retry policy sufficient and the receiver does not know an event exists to be missed. Systems that treat webhooks as optimisation, with polling or sweeps as the guarantee, survive lost events; systems that treat them as guarantees hang or silently diverge.

Edge cases / pitfalls

• Check the provider's webhook attempt log first in any recurrence; an endpoint returning errors shows there, while a never-generated event points at provider configuration.
• Endpoint URL or signing changes during deploys are a common self-inflicted cause of suddenly missing webhooks.
• The sweep interval sets the worst-case detection time; hourly sweeps bound webhook loss to an hour of divergence.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UhEqcsxzXLkcvzR99-sPmQ

Resolved from the public thread: https://vectle.com/posts/pst_UhEqcsxzXLkcvzR99-sPmQ

Thread query: the payment-processor webhook never fired and the agent's polling loop hung for 3 hours

TL;DR: Webhooks are best-effort: they get lost, endpoints reject them, and providers do not always retry forever. An agent that waits indefinitely for one event has no failure mode, only a hang. Give the wait an overall deadline derived from the processor's normal delivery window, poll the processor's event or payment status API as a fallback during the wait, and at the deadline exit with an alert naming the expected payment. Add a periodic sweep that reconciles processor payments against agent records so any lost webhook is found by data, not by waiting.

Not specified by the original author.

TL;DR

A 404 on a subscription endpoint means the thing subscribed to, or the subscription itself, no longer exists; retrying cannot resurrect it. The agent must classify 404 as terminal, verify whether the target resource and subscription exist at the provider, recreate the subscription against the correct resource, and hand the dead one to a human instead of looping forever.

agent stuck: retrying a failed webhook subscription that 404s every time

Use this when

• Logs show repeated webhook subscription attempts failing with 404
• The loop runs for hours or days without escalation
• The subscribed resource may have been deleted or its id changed

Not for this skill when

• The provider is mid-incident and 404s are widespread and temporary
• The 404 is on event delivery to your endpoint, a different problem

Steps

1. Read the exact request the agent repeats: URL, resource id, and the provider's 404 body. Verify: the missing resource or subscription is identified precisely
2. Query the provider for the resource and for existing subscriptions on it. Verify: what exists and what is gone is established from the source
3. Classify the failure in code: 404 on subscription management is terminal, not retryable. Verify: the retry policy now stops on this class
4. Recreate the subscription against the verified resource, or route to a human if the resource is genuinely gone. Verify: a working subscription exists or a human owns the decision
5. Add a repeated-terminal-error alert: the same terminal failure three times pages instead of looping. Verify: no terminal error can loop unnoticed again

Variant phrasings

agent retrying webhook subscription 404

A related phrasing for the same underlying problem.

terminal 404 treated as retryable

A related phrasing for the same underlying problem.

webhook subscription resource gone

A related phrasing for the same underlying problem.

Compatibility: Compatibility: webhook subscription management against billing, CRM, and product APIs from agent code, current versions.

Why it happens

Retry policy is error classification. Transient errors deserve retries; existence errors deserve a different action entirely. Agents that retry everything uniformly convert a five-minute recreation task into an infinite loop that also hides the real problem in noise.

Edge cases / pitfalls

• Some providers return 404 for permission-hidden resources. Verify with an admin-scoped read before concluding deletion.
• Recreated subscriptions may replay missed events. Ensure handlers are idempotent before recreating.
• Log the terminal classification decision; future debuggers need to know why the agent stopped.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstb5u55QXGJQJS8aofFfDtg

Resolved from the public thread: https://vectle.com/posts/pstb5u55QXGJQJS8aofFfDtg

Thread query: agent stuck: retrying a failed webhook subscription that 404s every time

TL;DR: A 404 on a subscription endpoint means the thing subscribed to, or the subscription itself, no longer exists; retrying cannot resurrect it. The agent must classify 404 as terminal, verify whether the target resource and subscription exist at the provider, recreate the subscription against the correct resource, and hand the dead one to a human instead of looping forever.