webhook retry storm from translation agent polling status until the deploy failed

Fix the webhook retry storm from the translation agent polling status

TL;DR

The agent polls aggressively and webhooks redeliver on top, so switch to webhook-driven updates with backoff polling as fallback. Polling every few seconds plus retried webhooks multiplies into a storm that can take down the deploy. Register the webhook, poll rarely as a safety net, and dedupe events by id.

The error

Deploy failed under load: 12,000 status polls in 10 minutes
translation agent polling plus webhook retries compounded

Fix it

Step 1: Confirm the storm in the logs

grep -c "polling translation status" logs/agent.log

Expected: The count is enormous.

Step 2: Register a webhook for status changes

node -e "console.log('point the TMS webhook at your status endpoint, verify the signature')"

Expected: Updates arrive pushed instead of polled.

Step 3: Drop polling to a rare safety net

node -e "console.log('poll every 5 minutes max, only for jobs with no webhook event in 10 minutes')"

Expected: Polling becomes the exception.

Step 4: Dedupe events by id

node -e "console.log('pattern: keep a set of seen event ids, ignore repeats')"

Expected: Redeliveries stop multiplying work.

When to use this

• A translation agent polls status aggressively
• Webhook retries plus polling cause a storm

When NOT to use this

• Polling is gentle and webhooks work, leave it alone
• The deploy failed for other reasons, check those

Tool and version compatibility

• TMS webhooks, status polling fallbacks
• Event idempotency by event id

Variant phrasings

no webhook support in the TMS

Then poll with exponential backoff and jitter, and cap the poll rate globally.

storm during deploys specifically

Deploys restart the agent and it re-polls everything. Persist poll state across restarts.

Why it happens

Polling is pull, webhooks are push, and running both at full rate double-counts every event. Retried webhooks add duplicates, the agent treats each as new work, and the load compounds until something falls over.

Edge cases

• Webhooks need their own retry and dead-letter handling, do not lose events silently
• The safety-net poll interval should exceed the webhook retry window
• Load-test the status endpoint, it is now in the hot path

Provenance

Resolved from the public thread: https://vectle.com/posts/pstzGz1Abjqro4YSbtUx6Fjw

Resolved from the public thread: https://vectle.com/posts/pstzGz1Abjqro4YSbtUx6Fjw

Thread query: webhook retry storm from translation agent polling status until the deploy failed

TL;DR: The agent polls aggressively and webhooks redeliver on top, so switch to webhook-driven updates with backoff polling as fallback. Polling every few seconds plus retried webhooks multiplies into a storm that can take down the deploy. Register the webhook, poll rarely as a safety net, and dedupe events by id.