agent's webhook handler processed events synchronously - slow handlers timed out and the provider retried,...

TL;DR

Acknowledge the webhook fast, then process the event asynchronously, and make processing idempotent. The provider redelivers anything it thinks timed out, so slow inline handlers guarantee duplicates. Return 200 in milliseconds, do the work in the background.

agent's webhook handler processed events synchronously  -  slow handlers timed out and the provider retried, double-processing orders

Steps

1. Move the work out of the request handler: verify the signature, enqueue the event (queue, job table, message bus), and return 200 immediately. Expected: handler response time drops to well under the provider's timeout.
2. Process events from the queue in a worker. Expected: slow processing no longer blocks the HTTP response.
3. Make processing idempotent using the provider's event id: record it before acting, skip events already seen. Expected: a redelivered event is a no-op the second time.
4. Return 200 only for events you accepted; return an error status only for things that will never succeed (bad signature, malformed body). Expected: the provider stops redelivering successfully-processed events.
5. Replay one historical event id through the worker twice and confirm a single effect. Expected: duplicates are now harmless.
6. Watch delivery metrics for a day: redelivery rate should fall and no new duplicates should appear. Expected: the double-processing stops entirely.

Use this when

• Orders, charges, or records get processed twice after webhook deliveries.
• The handler does real work (DB writes, API calls) before responding.
• Redeliveries arrive while the first attempt is still running.

Not for this skill when

• Events never reach your endpoint - that is a registration or routing problem.
• Signature verification fails - see the signature skills.
• Duplicates come from your own polling loop, not provider redelivery.

Variant phrasings

• webhook handler timed out and provider retried causing double orders
• synchronous webhook processing led to duplicate event handling
• slow webhook endpoint got redeliveries and double-processed

Why it happens

Webhook providers pick a timeout (often 5-30 seconds) and redeliver on timeout. A handler that processes synchronously almost always exceeds that under load, so the provider sends the event again while the first attempt is still working. Two in-flight attempts means two side effects.

Edge cases

• Providers that consider non-2xx as retry-worthy: never return 500 for a poison event you have already decided to skip; log it and return 200.
• Out-of-order delivery: an older event can arrive after a newer one; make state transitions tolerate that.
• Exactly-once is a myth with webhooks; design for at-least-once plus idempotent consumers from the start.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pj8La4mvsWD4PeaA8g8V7g