workerd error: queue message retry limit exceeded, message dropped

TL;DR

A queue message that keeps failing is retried up to the configured max, then dropped. Without a dead-letter queue, the failure is silent and the work is lost. Configure a dead-letter queue on the consumer, make the handler distinguish transient from permanent failures, and alert on dead-letter arrivals.

The query

workerd error: queue message retry limit exceeded, message dropped

Use this when

• Messages disappear after repeated consumer failures.
• The consumer logs show the same message failing N times then silence.
• You need to know WHY messages die, not just that they die.

Not for

• 'queue binding consumer not found' (binding config problem).
• Messages never delivered at all (producer or routing problem).
• First-attempt crashes (fix the handler bug directly).

Steps

Step 1: Confirm messages are being dropped, not delayed

Check the queue's metrics: failed and dropped counts climbing while the consumer reports no success.

Expected result: the drop is real. Retries are exhausted, not still in flight.

Step 2: Attach a dead-letter queue

[[queues.consumers]]
queue = "jobs"
dead_letter_queue = "jobs-dlq"
max_retries = 5

Expected result: exhausted messages land in jobs-dlq instead of vanishing. The original payload is preserved for inspection.

Step 3: Classify failures in the handler

async function handleBatch(batch) {
  const promises = batch.messages.map(async function(msg) {
    try {
      await handle(msg.body);
      msg.ack();
    } catch (e) {
      if (isPermanent(e)) {
        console.error("permanent failure, sending to DLQ", msg.id);
        msg.retry({ delaySeconds: 0 });
      } else {
        msg.retry();
      }
    }
  });
  await Promise.all(promises);
}

Expected result: transient errors (rate limits, brief outages) retry; permanent errors (bad payload, missing record) stop burning retries.

Step 4: Alert on the dead-letter queue

Run a consumer on the DLQ (or a cron poller) that pages when messages arrive.

Expected result: every dropped message becomes a visible incident with its payload attached, instead of a silent gap.

Step 5: Replay after fixing

Expected result: fixed messages can be redriven from the DLQ back into the main queue. The failure is recoverable, not terminal.

Why it happens

Retry limits exist to stop poison messages from wedging the consumer forever. But 'dropped' defaults to invisible: the queue did its job (gave up cleanly) and the application lost the work silently. The dead-letter queue is the missing half of the design; without it, observability ends at the retry count.

Edge cases

• Ack means 'done', even when the work is not done. Never ack a failed message unless a DLQ or log captured it.
• Batch semantics: one poison message can delay the whole batch. Handle per-message, not per-batch.
• max_retries counts across redeliveries; a consumer that crashes mid-batch replays the whole batch and burns retries fast.
• DLQ consumers need their own handling discipline or they can loop poison messages between queues.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_v99XMbVLjqWJEPwrId98LQ