# Fix webhook endpoint failing with 503 and causing a retry storm

## TL;DR
A 503ing webhook endpoint plus provider retries equals a retry storm that keeps the endpoint down. Return 200 immediately and process events asynchronously, then fix whatever is actually 503ing. Shedding load at the edge breaks the storm cycle.

## The error
```text
Webhook endpoint failing
HTTP 503 Service Unavailable. Provider retrying delivery; retry volume increasing.
```

## Use this skill
Agent? Search Vectle for this error:
```bash
curl -s "https://vectle.com/api/v1/search?q=webhook endpoint failed 503 retry storm"
```

## Fix it

### Step 1: Acknowledge fast, process later

```bash
Change the handler to validate, enqueue the event, and return 200 within a second. Move real work to a background worker.
```

Expected: Response times drop and the provider stops escalating retries.

### Step 2: Find the real cause of the 503

```bash
Check your app logs and infra metrics for what is actually failing: DB overload, downstream timeout, or crashed workers.
```

Expected: You have the root cause, not just the symptom.

### Step 3: Fix the underlying failure

```bash
Scale the bottleneck, fix the downstream call, or restart the failed dependency.
```

Expected: The endpoint stops 503ing under normal load.

### Step 4: Drain the retry backlog

```bash
Let the provider retries flow through the fast-ack endpoint; workers chew through the queue.
```

Expected: The queue drains and retry volume returns to baseline.

### Step 5: Add idempotency to the worker

```bash
Key processing on the provider's event id so retried deliveries do not double-apply.
```

Expected: Duplicate deliveries become harmless.

## When this applies

- Webhook endpoints 503 and providers keep retrying
- Retry volume is growing and keeping the service down
- You are designing webhook receivers for scale

## When it doesn't

- The endpoint 404s (check the URL registration)
- The endpoint 401s (check the auth or signature)
- A single event fails (that is handler logic, not a storm)

## Compatibility

Webhook receivers generally: Stripe, Shopify, GitHub, and others. Any stack.

## Variant phrasings

### webhook retry storm 503

Same pattern. Fast ack plus async processing is the standard fix.

### webhook endpoint overloaded retries

Overload from retries is self-inflicted; shedding at the edge is the circuit breaker.

### provider retrying webhooks exponentially

Exponential backoff still overwhelms a down endpoint eventually. Fix the endpoint, not the backoff.

## Why it happens

Providers retry failed deliveries with backoff, which is fine for transient blips. But when the endpoint 503s because it does heavy work inline, each retry adds load, which causes more 503s, which causes more retries. The storm is a feedback loop, and only acknowledging fast breaks it.

## Edge cases

- Some providers give up after N retries; reconcile missed events from their API after recovery
- A poison event that always 503s will storm forever; dead-letter it after a few tries
- Autoscaling on webhook traffic can mask the real bottleneck; fix the slow path first

## If it still fails

- Reproduce with a test event from the provider dashboard to separate delivery problems from handler bugs.
- Log the raw payload shape, never customer PII, so the next failure is comparable.
- Check the provider status page; delivery outages mimic endpoint bugs.
- Replay a known-good event after the fix to prove the path works, not just that errors stopped.
- If signature failures persist with correct code, rotate the signing secret once; stale secrets cause silent mismatches.

## Prevention

- Return 200 fast and process async on every new webhook receiver.
- Make event handling idempotent from day one; retries are guaranteed.
- Monitor delivery success rates, not just endpoint uptime.
- Keep signing secrets per endpoint and rotate them on a schedule.
- Reconcile critical events against the provider API, not just webhooks.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QzfrlsPERSBr-IUu0-I2kQ