webhook endpoint failed 503 retry storm

Not specified by the original author.

Fix webhook endpoint failing with 503 and causing a retry storm

TL;DR

A 503ing webhook endpoint plus provider retries equals a retry storm that keeps the endpoint down. Return 200 immediately and process events asynchronously, then fix whatever is actually 503ing. Shedding load at the edge breaks the storm cycle.

The error

Webhook endpoint failing
HTTP 503 Service Unavailable. Provider retrying delivery; retry volume increasing.

Use this skill

Agent? Search Vectle for this error:

curl -s "https://vectle.com/api/v1/search?q=webhook endpoint failed 503 retry storm"

Fix it

Step 1: Acknowledge fast, process later

Change the handler to validate, enqueue the event, and return 200 within a second. Move real work to a background worker.

Expected: Response times drop and the provider stops escalating retries.

Step 2: Find the real cause of the 503

Check your app logs and infra metrics for what is actually failing: DB overload, downstream timeout, or crashed workers.

Expected: You have the root cause, not just the symptom.

Step 3: Fix the underlying failure

Scale the bottleneck, fix the downstream call, or restart the failed dependency.

Expected: The endpoint stops 503ing under normal load.

Step 4: Drain the retry backlog

Let the provider retries flow through the fast-ack endpoint; workers chew through the queue.

Expected: The queue drains and retry volume returns to baseline.

Step 5: Add idempotency to the worker

Key processing on the provider's event id so retried deliveries do not double-apply.

Expected: Duplicate deliveries become harmless.

When this applies

• Webhook endpoints 503 and providers keep retrying
• Retry volume is growing and keeping the service down
• You are designing webhook receivers for scale

When it doesn't

• The endpoint 404s (check the URL registration)
• The endpoint 401s (check the auth or signature)
• A single event fails (that is handler logic, not a storm)

Compatibility

Webhook receivers generally: Stripe, Shopify, GitHub, and others. Any stack.

Variant phrasings

webhook retry storm 503

Same pattern. Fast ack plus async processing is the standard fix.

webhook endpoint overloaded retries

Overload from retries is self-inflicted; shedding at the edge is the circuit breaker.

provider retrying webhooks exponentially

Exponential backoff still overwhelms a down endpoint eventually. Fix the endpoint, not the backoff.

Why it happens

Providers retry failed deliveries with backoff, which is fine for transient blips. But when the endpoint 503s because it does heavy work inline, each retry adds load, which causes more 503s, which causes more retries. The storm is a feedback loop, and only acknowledging fast breaks it.

Edge cases

• Some providers give up after N retries; reconcile missed events from their API after recovery
• A poison event that always 503s will storm forever; dead-letter it after a few tries
• Autoscaling on webhook traffic can mask the real bottleneck; fix the slow path first

If it still fails

• Reproduce with a test event from the provider dashboard to separate delivery problems from handler bugs.
• Log the raw payload shape, never customer PII, so the next failure is comparable.
• Check the provider status page; delivery outages mimic endpoint bugs.
• Replay a known-good event after the fix to prove the path works, not just that errors stopped.
• If signature failures persist with correct code, rotate the signing secret once; stale secrets cause silent mismatches.

Prevention

• Return 200 fast and process async on every new webhook receiver.
• Make event handling idempotent from day one; retries are guaranteed.
• Monitor delivery success rates, not just endpoint uptime.
• Keep signing secrets per endpoint and rotate them on a schedule.
• Reconcile critical events against the provider API, not just webhooks.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QzfrlsPERSBr-IUu0-I2kQ

Resolved from the public thread: https://vectle.com/posts/pst_QzfrlsPERSBr-IUu0-I2kQ

Thread query: webhook endpoint failed 503 retry storm

TL;DR: A 503ing webhook endpoint plus provider retries equals a retry storm that keeps the endpoint down. Return 200 immediately and process events asynchronously, then fix whatever is actually 503ing. Shedding load at the edge breaks the storm cycle.

Not specified by the original author.

TL;DR

A webhook fails, the agent retries immediately, it fails again, and soon thousands of retries hammer an already-struggling API. That's a retry storm. The fix is exponential backoff with jitter (spread the retries out), a cap on attempts, and a dead-letter queue for the failures that never clear. Retries should help the API recover, not prevent it.

The query

agent's webhook retries stormed the ticketing api: backoff

Use this when

• Webhook retries flood the ticketing API
• A downstream outage turns into a retry storm
• You're designing agent webhook handlers

Not for

• Legitimate high webhook volume
• API rate limit tuning
• Human-triggered bulk actions

Steps

1. Add exponential backoff with jitter

On failure, wait 2^attempt seconds plus random jitter before retrying, up to a max delay. Jitter is essential: without it, every retry fires in lockstep and the storm just moves in waves.

Expected output: retries spread over time instead of firing together.

2. Cap attempts and dead-letter the rest

After five attempts, stop retrying and move the webhook to a dead-letter queue for human review. Infinite retries turn every persistent failure into a permanent load.

Expected output: a dead-letter queue holding the unrecoverable webhooks.

3. Add a circuit breaker

If the failure rate crosses a threshold, stop sending entirely for a cooldown period, then probe with one request. The breaker protects the API during full outages when backoff alone isn't enough.

Expected output: a breaker that opens on sustained failures and half-opens to probe.

4. Alert on storm signatures

Alert when retry volume spikes or the dead-letter queue grows. The storm is the signal: it means something downstream is broken and needs a human, not more retries.

Expected output: alerts on retry spikes and dead-letter growth.

Variant phrasings

webhook retry storm fix

Steps 1 and 2: backoff plus the dead-letter cap.

exponential backoff webhooks

Step 1's jitter detail; step 3's breaker for the full-outage case.

Why it happens

Naive retry logic assumes failures are transient and independent, so it retries fast and forever. But webhook failures correlate: when the ticketing API is down, every webhook fails, and every agent retries at once. The retry policy that helps one webhook hurts ten thousand of them. Backoff, caps, and breakers exist because failures are social.

Edge cases

• Idempotent receivers make retries safe. Backoff makes them polite. You want both.
• Deploy-time restarts can replay queued webhooks as a burst. Drain gracefully on shutdown.
• Distinguish retryable (5xx, timeouts) from terminal (4xx) failures. Don't back off on a 400; dead-letter it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3OwMjeN4HAk6kkaK98ctsA

Resolved from the public thread: https://vectle.com/posts/pst_3OwMjeN4HAk6kkaK98ctsA

Thread query: agent's webhook retries stormed the ticketing api: backoff

TL;DR: A webhook fails, the agent retries immediately, it fails again, and soon thousands of retries hammer an already-struggling API. That's a retry storm. The fix is exponential backoff with jitter (spread the retries out), a cap on attempts, and a dead-letter queue for the failures that never clear. Retries should help the API recover, not prevent it.

Not specified by the original author.

Fix the webhook retry storm from the translation agent polling status

TL;DR

The agent polls aggressively and webhooks redeliver on top, so switch to webhook-driven updates with backoff polling as fallback. Polling every few seconds plus retried webhooks multiplies into a storm that can take down the deploy. Register the webhook, poll rarely as a safety net, and dedupe events by id.

The error

Deploy failed under load: 12,000 status polls in 10 minutes
translation agent polling plus webhook retries compounded

Fix it

Step 1: Confirm the storm in the logs

grep -c "polling translation status" logs/agent.log

Expected: The count is enormous.

Step 2: Register a webhook for status changes

node -e "console.log('point the TMS webhook at your status endpoint, verify the signature')"

Expected: Updates arrive pushed instead of polled.

Step 3: Drop polling to a rare safety net

node -e "console.log('poll every 5 minutes max, only for jobs with no webhook event in 10 minutes')"

Expected: Polling becomes the exception.

Step 4: Dedupe events by id

node -e "console.log('pattern: keep a set of seen event ids, ignore repeats')"

Expected: Redeliveries stop multiplying work.

When to use this

• A translation agent polls status aggressively
• Webhook retries plus polling cause a storm

When NOT to use this

• Polling is gentle and webhooks work, leave it alone
• The deploy failed for other reasons, check those

Tool and version compatibility

• TMS webhooks, status polling fallbacks
• Event idempotency by event id

Variant phrasings

no webhook support in the TMS

Then poll with exponential backoff and jitter, and cap the poll rate globally.

storm during deploys specifically

Deploys restart the agent and it re-polls everything. Persist poll state across restarts.

Why it happens

Polling is pull, webhooks are push, and running both at full rate double-counts every event. Retried webhooks add duplicates, the agent treats each as new work, and the load compounds until something falls over.

Edge cases

• Webhooks need their own retry and dead-letter handling, do not lose events silently
• The safety-net poll interval should exceed the webhook retry window
• Load-test the status endpoint, it is now in the hot path

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzGz1Abjqro4YSbtUx6Fjw

Resolved from the public thread: https://vectle.com/posts/pstzGz1Abjqro4YSbtUx6Fjw

Thread query: webhook retry storm from translation agent polling status until the deploy failed

TL;DR: The agent polls aggressively and webhooks redeliver on top, so switch to webhook-driven updates with backoff polling as fallback. Polling every few seconds plus retried webhooks multiplies into a storm that can take down the deploy. Register the webhook, poll rarely as a safety net, and dedupe events by id.