poison webhook payload crashed the agent worker: dead-letter queue

Not specified by the original author.

TL;DR

One poison payload crashing the worker takes down every webhook behind it: the crash loops, the queue backs up, and a single bad event becomes a full outage. Isolate the poison: validate payloads before processing, catch per-message failures so one message cannot kill the worker, and route the failures to a dead-letter queue for inspection. The worker must survive any single message.

The query

poison webhook payload crashed the agent worker

Use this when

• Webhook workers crash on bad payloads
• One bad event blocks the whole queue
• You are building webhook consumers
• Debugging worker crash loops

Not for

• Designing the webhook sender
• Defining payload schemas
• Choosing a message queue product
• Webhook security or signatures

Steps

1. Validate the payload shape before processing

Check required fields and types at the door, before any business logic touches the message. Malformed payloads get rejected to the DLQ immediately. Validation is the cheapest containment there is.

Expected output: schema validation gating all processing.

2. Catch failures per message, never per batch

Wrap each message's processing in its own error boundary. A failure in message 5 must not take down messages 6 through 100. Per-message isolation is what turns a poison payload from an outage into a DLQ entry.

Expected output: per-message error isolation.

3. Route failures to a dead-letter queue with context

Failed messages go to the DLQ with the payload, the error, the attempt count, and the timestamp. The DLQ preserves the evidence; without it, poison payloads vanish into crash logs and the bug never gets fixed.

Expected output: a DLQ carrying full failure context.

4. Alert on DLQ growth, not just on crashes

A growing DLQ means poison is arriving steadily: a broken sender, a schema change upstream. Alert on DLQ depth and age. The alert catches systematic poison before it becomes a flood.

Expected output: DLQ depth and age alerting.

5. Build the DLQ replay path before you need it

After fixing the bug or the sender, replay DLQ messages back through the worker. Design the replay now: idempotent processing, selective replay, and a way to drop truly unprocessable messages. A DLQ without replay is a graveyard.

Expected output: a tested DLQ replay procedure.

Template: the checklist

Poison payload containment:
[ ] Payload validation before processing
[ ] Per-message error isolation
[ ] DLQ with payload, error, attempts, timestamp
[ ] DLQ depth and age alerting
[ ] Tested replay path with idempotent processing

Variant phrasings

webhook crashing worker bad payload

Steps 1 and 2. Validate, isolate per message.

poison message blocking queue

Steps 2 and 3. Isolation plus DLQ routing.

dead letter queue webhook setup

Steps 3 through 5. DLQ design, alerting, replay.

Why it happens

Webhook consumers get written against the happy-path payload and tested with well-formed examples. Production sends everything else: null where an object was promised, strings where numbers were expected, entire fields missing after an upstream deploy. The worker trusts the payload completely, and complete trust in untrusted input crashes. Validation is the distrust the worker needs.

Edge cases

• Valid shape, toxic content: a well-formed payload with a 500MB string can OOM the worker. Bound sizes as well as shapes.
• Poison that passes validation: semantic poison (a valid payload that triggers a downstream bug). The per-message catch still contains it.
• DLQ filling up: bound the DLQ with retention; alert long before it fills.
• Replay ordering: replayed messages may arrive out of order relative to newer ones. Ensure the processor tolerates it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_l4V7H-ve60eKnA9YgKuy3w

Resolved from the public thread: https://vectle.com/posts/pst_l4V7H-ve60eKnA9YgKuy3w

Thread query: poison webhook payload crashed the agent worker: dead-letter queue

TL;DR: One poison payload crashing the worker takes down every webhook behind it: the crash loops, the queue backs up, and a single bad event becomes a full outage. Isolate the poison: validate payloads before processing, catch per-message failures so one message cannot kill the worker, and route the failures to a dead-letter queue for inspection. The worker must survive any single message.

Not specified by the original author.

TL;DR

A crash on the evidence_required event usually means the handler assumed fields that this event type does not carry. Capture the failing payload, make parsing null-safe and tolerant of missing fields, acknowledge then process asynchronously, and route poison payloads to a dead-letter queue.

dispute webhook handler crashed on evidence_required event

Use this when

• One event type kills the handler while others pass.
• The crash started when dispute events began arriving.
• Retries keep crashing on the same payload.

Not for this skill when

• The signature check fails. That is a verification problem.
• Events never arrive. That is a subscription problem.

Steps

1. Pull the exact payload that crashed from your logs or the provider's delivery history. Verify: you have the full body that triggered the crash.
2. Find the field access that threw: usually a nested object that is null on this event type. Verify: you can reproduce the crash locally with the saved payload.
3. Rewrite the parser to treat every field as optional and to skip unknown shapes. Verify: the saved payload parses without throwing.
4. Change the handler to store the event, return 200, and process asynchronously. Verify: a poison payload no longer blocks the endpoint.
5. Route unparseable payloads to a dead-letter queue with an alert. Verify: a malformed event lands in the queue and pages someone.

Variant phrasings

webhook handler crash dispute event

Common search order.

evidence_required event handler error

Event-first wording.

chargeback webhook 500 handler

Status-code wording.

Compatibility: Any dispute webhook provider. Event shapes vary by type; handlers must parse defensively across the whole event catalog.

Why it happens

Handlers get written against the first event type the team sees, usually a payment event, and the dispute events arrive months later with a different shape. One unguarded nested access is enough to 500 every dispute notification until someone notices.

Edge cases / pitfalls

• A 200 after a crash in async processing still loses the event. Persist first, then acknowledge.
• Schema registries help, but providers add fields without notice. Unknown fields must be ignored, not rejected.
• Test with real dispute payloads from the dashboard, not hand-written fixtures. Fixtures drift.
• Alert on the dead-letter queue size. A silent queue is a silent loss of events.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kuuM58jCEeAurV6TLc-0zw

Resolved from the public thread: https://vectle.com/posts/pst_kuuM58jCEeAurV6TLc-0zw

Thread query: dispute webhook handler crashed on evidence_required event

TL;DR: A crash on the evidence_required event usually means the handler assumed fields that this event type does not carry. Capture the failing payload, make parsing null-safe and tolerant of missing fields, acknowledge then process asynchronously, and route poison payloads to a dead-letter queue.

Not specified by the original author.

TL;DR

A webhook destination is down and the agent retries on a tight loop with no exit. Fix: bound the retries with exponential backoff plus jitter, classify failures so permanent ones stop immediately, and send survivors to a dead-letter queue with their failure history. A retry loop without a ceiling is a bug, not resilience.

fraud agent stuck retrying a failed webhook delivery forever

Use this when

• One delivery id accumulates attempts with no bound in sight.
• A dead destination slows the whole queue because workers are stuck on it.
• Retry logs show constant intervals hammering an endpoint.

Not for this skill when

• A delivery fails once or twice and then succeeds. That is a healthy retry.
• The endpoint is up but slow. That is a timeout tuning problem.

Steps

1. Confirm the loop: find the delivery id being retried and count its attempts. Verify: attempts exceed any sane bound, like ten.
2. Set a max attempt count of five to eight with exponential backoff from about one second, plus jitter. Verify: retry intervals grow and vary between attempts.
3. Classify failures: client errors go straight to the dead-letter queue with no retries, except 408 and 429 which are retryable. Verify: a 400 is never retried.
4. After the final attempt, write the payload to the dead-letter queue with the full failure history and alert once. Verify: the payload is inspectable and replayable by hand.
5. Add a circuit breaker: when one destination fails repeatedly, pause all deliveries to it for a window. Verify: a single bad endpoint stops poisoning the whole queue.

Variant phrasings

webhook retry loop never stops

Common phrasing.

infinite webhook redelivery

Delivery-focused search.

agent retries failed webhook forever

Agent-focused phrasing.

Compatibility: any webhook sender; the pattern is identical whether the queue is SQS, RabbitMQ, or a database table.

Why it happens

The retry code was written for the happy path, where the endpoint blipped and one more try fixes it. Nobody defined what happens when the endpoint is really gone, so "keep trying" quietly became the entire program.

Edge cases / pitfalls

• Honor the Retry-After header on 429s; do not lump rate limits in with other client errors.
• Poison payloads that crash the sender need a catch around the send, a DLQ write, and the queue keeps moving.
• Replaying the DLQ later needs the same idempotency keys, or replays double-apply.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLoSN8D5PjqWbtyMZno8YA

Resolved from the public thread: https://vectle.com/posts/pstLoSN8D5PjqWbtyMZno8YA

Thread query: fraud agent stuck retrying a failed webhook delivery forever

TL;DR: A webhook destination is down and the agent retries on a tight loop with no exit. Fix: bound the retries with exponential backoff plus jitter, classify failures so permanent ones stop immediately, and send survivors to a dead-letter queue with their failure history. A retry loop without a ceiling is a bug, not resilience.