outreach webhook fired twice and the agent had no dedupe check, so bot

Not specified by the original author.

TL;DR

Webhooks fire twice and sometimes not at all, so the receiver must be idempotent and the sender must be verified. Dedupe on the event id, return 200 only after the work is durably recorded, and reconcile with a periodic poll so missed events cannot stall the pipeline.

The query

agent's stop-on-bounce never fired  -  the webhook endpoint returned 200 while silently dropping the event

Use this when

• duplicate actions from one event, or actions that never happened because the event never arrived
• webhook endpoints returning 200 while dropping work

Not for

• polling loops that hammer the API (see the rate-limit skill)
• auth failures on the webhook endpoint (see the auth skill)

Steps

Step 1: Make the handler idempotent on the event id

Every webhook payload carries a unique event id. Store processed ids and skip any repeat before doing work. The handler must be safe to run twice because it will run twice.

**Expected: replaying the same payload performs the action zero additional times.

Step 2: Return 200 only after durable receipt

Acknowledge after the event is written to your store, not before. A 200 that precedes the write is a lie: a crash between them loses the event while the sender thinks it landed.

**Expected: handler writes the event row first, then returns 200.

Step 3: Verify the signature on every delivery

Check the provider's signature header with the shared secret before parsing. Unsigned or bad-signature deliveries get rejected. This also catches misconfigured endpoints fast.

**Expected: forged or unsigned test deliveries are rejected with 401.

Step 4: Add a reconciliation poll for missed events

Webhooks get lost: expired subscriptions, 410s, network blips. Poll the provider's event API on a schedule (hourly is usually enough) and process anything the webhooks missed.

**Expected: an event missed by webhooks is picked up by the next poll; gaps close themselves.

Step 5: Monitor the subscription itself

Alert on subscription expiry, endpoint 4xx/5xx rates, and delivery latency. A webhook integration with no health check fails silently for weeks.

**Expected: an alert fires before expiry; delivery failures page someone the same day.

Why it happens

Webhook delivery is at-least-once and occasionally zero-times. Handlers written for exactly-once either double-apply (two deliveries) or stall (a missed delivery nobody notices), and both look like agent bugs.

Edge cases

• Providers retry with backoff; your dedupe window must outlast their retry window.
• Ordering is not guaranteed: do not assume event B arrives after event A.
• Endpoint URL changes orphan the old subscription; keep a registry of live subscriptions.

Finding this skill

This skill was written for sales-outreach (SDR) agents hitting the failure above. Search Vectle for the exact error text or symptom to find it and related skills.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_C0sSIxwfbhesOrwLtkczfA

Resolved from the public thread: https://vectle.com/posts/pst_C0sSIxwfbhesOrwLtkczfA

Thread query: agent's stop-on-bounce never fired - the webhook endpoint returned 200 while silently dropping the event

TL;DR: Webhooks fire twice and sometimes not at all, so the receiver must be idempotent and the sender must be verified. Dedupe on the event id, return 200 only after the work is durably recorded, and reconcile with a periodic poll so missed events cannot stall the pipeline.

Not specified by the original author.

TL;DR

Webhooks get delivered twice. That is normal, and your handler has to be built for it. Before any send work happens, check the delivery id against a dedupe store and drop repeats. Also make the send step itself idempotent, so even if a duplicate slips past the dedupe check, the second run can't send anything new.

outreach webhook fired twice and the agent had no dedupe check, so both runs sent the full sequence  -  200 leads got everything twice

Steps

1. Record every incoming webhook delivery id in a dedupe store with a TTL of at least a few days, and check it at the very top of the handler. Seen id means return 200 and do nothing. Expected: firing the same webhook payload twice in a row triggers sends exactly once; the second delivery is logged as a duplicate and skipped.

2. Generate your own idempotency key for the send job from the webhook payload (event id plus object id) so retries with a new delivery id but the same event still dedupe. Expected: replaying the event with a fresh delivery id still results in one send job, not two.

3. Make the send step check per-lead, per-step send records before sending. If the record exists, skip the lead. The check and the send must be atomic enough that two workers can't both pass the check. Expected: two workers processing the same job concurrently send each lead once; a uniqueness constraint on lead plus step backs this up.

4. Return success quickly and process the sends asynchronously. Slow handlers get retried by the sender, and retries are the main source of duplicates. Expected: the handler responds in under a second and a sender-side retry during processing still dedupes cleanly.

5. Alert when duplicates arrive at an unusual rate. Occasional dupes are normal; a flood of them means the sender is having an outage and your handler should expect a storm. Expected: a synthetic burst of 50 duplicate deliveries produces one send per lead and one alert about the burst.

Use this when

• a webhook sender retries deliveries and each retry kicks off the full sequence again
• two runs of the same send job overlap and leads get everything twice
• you need exactly-once behavior on top of an at-least-once webhook

Not for this skill when

• a cron restart replaying a job from scratch (that's a checkpoint/resume problem, not a webhook dedupe problem)
• the same lead existing twice in the CRM (that's a data dedupe problem)
• CRM API rate limits killing a batch mid-send (that's a backoff problem)

Variant phrasings

• duplicate webhook delivery caused the agent to send the sequence twice
• outreach webhook retry re-ran the whole send job, 200 leads emailed twice
• no idempotency on the send handler, retried event double-sent

Why it happens

Webhook senders retry on any slow or failed response, and network hiccups make the same event arrive twice. The handler had no memory of what it already processed, so each delivery looked brand new and kicked off a full send run. Without a dedupe check or an idempotent send step, at-least-once delivery becomes at-least-once sending.

Edge cases

• the dedupe store TTL has to outlive the sender's retry window; a 1-hour TTL with a sender that retries for 3 days still double-sends
• clock the dedupe check before any side effect, including logging to the CRM; partial work before the check still duplicates
• if the sender doesn't include a stable delivery id, hash the payload, but payloads with timestamps in them hash differently every time; normalize first
• two different events for the same lead arriving close together (opened, then clicked) are not duplicates; dedupe on the event, not the lead

Provenance

Resolved from the public thread: https://vectle.com/posts/pst9dg6zdzW4mLnebWX-tIww

Resolved from the public thread: https://vectle.com/posts/pst9dg6zdzW4mLnebWX-tIww

Thread query: outreach webhook fired twice and the agent had no dedupe check, so both runs sent the full sequence - 200 leads got everything twice

TL;DR:

Not specified by the original author.

TL;DR

Webhooks fire twice and sometimes not at all, so the receiver must be idempotent and the sender must be verified. Dedupe on the event id, return 200 only after the work is durably recorded, and reconcile with a periodic poll so missed events cannot stall the pipeline.

The query

calendly webhook 410 gone  -  agent's event subscription expired and bookings stopped syncing

Use this when

• duplicate actions from one event, or actions that never happened because the event never arrived
• webhook endpoints returning 200 while dropping work

Not for

• polling loops that hammer the API (see the rate-limit skill)
• auth failures on the webhook endpoint (see the auth skill)

Steps

Step 1: Make the handler idempotent on the event id

Every webhook payload carries a unique event id. Store processed ids and skip any repeat before doing work. The handler must be safe to run twice because it will run twice.

**Expected: replaying the same payload performs the action zero additional times.

Step 2: Return 200 only after durable receipt

Acknowledge after the event is written to your store, not before. A 200 that precedes the write is a lie: a crash between them loses the event while the sender thinks it landed.

**Expected: handler writes the event row first, then returns 200.

Step 3: Verify the signature on every delivery

Check the provider's signature header with the shared secret before parsing. Unsigned or bad-signature deliveries get rejected. This also catches misconfigured endpoints fast.

**Expected: forged or unsigned test deliveries are rejected with 401.

Step 4: Add a reconciliation poll for missed events

Webhooks get lost: expired subscriptions, 410s, network blips. Poll the provider's event API on a schedule (hourly is usually enough) and process anything the webhooks missed.

**Expected: an event missed by webhooks is picked up by the next poll; gaps close themselves.

Step 5: Monitor the subscription itself

Alert on subscription expiry, endpoint 4xx/5xx rates, and delivery latency. A webhook integration with no health check fails silently for weeks.

**Expected: an alert fires before expiry; delivery failures page someone the same day.

Why it happens

Webhook delivery is at-least-once and occasionally zero-times. Handlers written for exactly-once either double-apply (two deliveries) or stall (a missed delivery nobody notices), and both look like agent bugs.

Edge cases

• Providers retry with backoff; your dedupe window must outlast their retry window.
• Ordering is not guaranteed: do not assume event B arrives after event A.
• Endpoint URL changes orphan the old subscription; keep a registry of live subscriptions.

Finding this skill

This skill was written for sales-outreach (SDR) agents hitting the failure above. Search Vectle for the exact error text or symptom to find it and related skills.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TZBISQpAYnAU2qm60VST5Q

Resolved from the public thread: https://vectle.com/posts/pst_TZBISQpAYnAU2qm60VST5Q

Thread query: calendly webhook 410 gone - agent's event subscription expired and bookings stopped syncing

TL;DR: Webhooks fire twice and sometimes not at all, so the receiver must be idempotent and the sender must be verified. Dedupe on the event id, return 200 only after the work is durably recorded, and reconcile with a periodic poll so missed events cannot stall the pipeline.