agent's queue consumer test message poisoned the real queue: retry storm on production

TL;DR

The agent sent a test message to the production queue. The production consumer cannot process it, so it retries, fails, and retries again: a retry storm burning quota and delaying real messages. Purge or dead-letter the test message, verify the consumer drains, and keep test traffic on a test queue permanently.

The query

agent's queue consumer test message poisoned the real queue: retry storm on production

Use this when

• The production consumer retries the same message repeatedly.
• The message is recognizable as a test (test payload, test marker).
• Real messages are delayed behind the storm.

Not for

• Genuine production poison messages (same mechanics, but the fix is in the producer).
• Consumer code bugs failing on valid messages (fix the consumer).
• Retry storms with no test involvement (capacity or bug problem).

Steps

Step 1: Identify the poison message

Check the consumer logs and the dead-letter queue (if configured) for the looping message.

Expected result: the test message is identified by its payload or marker.

Step 2: Remove it from the production queue

Purge the specific message or move it to the dead-letter queue. Do not purge the whole queue unless the storm has fully wedged it.

Expected result: the retry loop stops. The consumer resumes processing real messages.

Step 3: Verify the consumer drains

Watch the queue depth fall to normal and confirm real messages process correctly.

Expected result: the storm is over and no backlog damage remains.

Step 4: Create a permanent test queue

A separate queue with a test consumer (or the same consumer code against test bindings) for all agent testing.

Expected result: test traffic has a home that is not production. The agent never needs the production queue for tests.

Step 5: Ban production-queue testing in the runbook

Expected result: the agent's test flow targets the test queue by construction. The production queue is not a test fixture.

Why it happens

Queues look like convenient test fixtures: send a message, watch the consumer. But the production consumer treats every message as real work, and malformed test messages become poison messages by definition. The retry machinery then amplifies one bad test into a production incident.

Edge cases

• Without a dead-letter queue, the poison message retries until max_retries then drops silently. Configure the DLQ so the next one is visible.
• The storm may have delayed or duplicated real messages. Audit downstream effects, not just the queue.
• Test markers in payloads help forensics. Require the agent to tag test messages even on the test queue.
• If the consumer lacks idempotency, the storm may have double-processed real messages interleaved with the poison. Check for that.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstVURboaLu76o6hXXQemD3Q