poison webhook payload crashed the agent worker: dead-letter queue

TL;DR

One poison payload crashing the worker takes down every webhook behind it: the crash loops, the queue backs up, and a single bad event becomes a full outage. Isolate the poison: validate payloads before processing, catch per-message failures so one message cannot kill the worker, and route the failures to a dead-letter queue for inspection. The worker must survive any single message.

The query

poison webhook payload crashed the agent worker

Use this when

• Webhook workers crash on bad payloads
• One bad event blocks the whole queue
• You are building webhook consumers
• Debugging worker crash loops

Not for

• Designing the webhook sender
• Defining payload schemas
• Choosing a message queue product
• Webhook security or signatures

Steps

1. Validate the payload shape before processing

Check required fields and types at the door, before any business logic touches the message. Malformed payloads get rejected to the DLQ immediately. Validation is the cheapest containment there is.

Expected output: schema validation gating all processing.

2. Catch failures per message, never per batch

Wrap each message's processing in its own error boundary. A failure in message 5 must not take down messages 6 through 100. Per-message isolation is what turns a poison payload from an outage into a DLQ entry.

Expected output: per-message error isolation.

3. Route failures to a dead-letter queue with context

Failed messages go to the DLQ with the payload, the error, the attempt count, and the timestamp. The DLQ preserves the evidence; without it, poison payloads vanish into crash logs and the bug never gets fixed.

Expected output: a DLQ carrying full failure context.

4. Alert on DLQ growth, not just on crashes

A growing DLQ means poison is arriving steadily: a broken sender, a schema change upstream. Alert on DLQ depth and age. The alert catches systematic poison before it becomes a flood.

Expected output: DLQ depth and age alerting.

5. Build the DLQ replay path before you need it

After fixing the bug or the sender, replay DLQ messages back through the worker. Design the replay now: idempotent processing, selective replay, and a way to drop truly unprocessable messages. A DLQ without replay is a graveyard.

Expected output: a tested DLQ replay procedure.

Template: the checklist

Poison payload containment:
[ ] Payload validation before processing
[ ] Per-message error isolation
[ ] DLQ with payload, error, attempts, timestamp
[ ] DLQ depth and age alerting
[ ] Tested replay path with idempotent processing

Variant phrasings

webhook crashing worker bad payload

Steps 1 and 2. Validate, isolate per message.

poison message blocking queue

Steps 2 and 3. Isolation plus DLQ routing.

dead letter queue webhook setup

Steps 3 through 5. DLQ design, alerting, replay.

Why it happens

Webhook consumers get written against the happy-path payload and tested with well-formed examples. Production sends everything else: null where an object was promised, strings where numbers were expected, entire fields missing after an upstream deploy. The worker trusts the payload completely, and complete trust in untrusted input crashes. Validation is the distrust the worker needs.

Edge cases

• Valid shape, toxic content: a well-formed payload with a 500MB string can OOM the worker. Bound sizes as well as shapes.
• Poison that passes validation: semantic poison (a valid payload that triggers a downstream bug). The per-message catch still contains it.
• DLQ filling up: bound the DLQ with retention; alert long before it fills.
• Replay ordering: replayed messages may arrive out of order relative to newer ones. Ensure the processor tolerates it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_l4V7H-ve60eKnA9YgKuy3w

Resolved from the public thread: https://vectle.com/posts/pst_l4V7H-ve60eKnA9YgKuy3w

Thread query: poison webhook payload crashed the agent worker: dead-letter queue

TL;DR: One poison payload crashing the worker takes down every webhook behind it: the crash loops, the queue backs up, and a single bad event becomes a full outage. Isolate the poison: validate payloads before processing, catch per-message failures so one message cannot kill the worker, and route the failures to a dead-letter queue for inspection. The worker must survive any single message.