calendly webhook 410 gone — agent's event subscription expire

Not specified by the original author.

TL;DR: Detect the 410 and automatically re-create the webhook subscription, then persist the new subscription ID. A 410 Gone on a Lever webhook subscription means that subscription is dead and will never deliver again - polling it or retrying the same subscription ID cannot recover. The agent needs a re-subscribe path, not a retry loop.

410 Gone - subscription expired

1. Confirm the subscription is actually dead: query the webhook subscription status in Lever or check the subscription management endpoint. Expected: the subscription shows as expired or missing.
2. Create a fresh webhook subscription with the same event types and target URL as the dead one. Expected: a new subscription ID is returned.
3. Persist the new subscription ID in the agent's config or state store, replacing the old one everywhere it is referenced. Expected: no code or config still points at the expired subscription ID.
4. Backfill the gap: pull events or changes since the subscription died (Lever's list endpoints support updated-at filters) so nothing that happened during the outage is lost. Expected: the missed window is covered by the backfill.
5. Add a health check that verifies the subscription is alive on a schedule (for example daily) and pages someone when it is not. Expected: the next expiry is caught by the check, not by a silent empty queue.

Use this when

• Lever webhook deliveries stop and the subscription answers 410 Gone
• The agent's event feed goes quiet with no errors in the agent's own logs
• A subscription was created long ago and nobody has touched it since

Not for this skill when

• Events arrive but fail signature verification (that is a secret problem, not an expired subscription)
• The subscription is alive but events are delayed (check Lever's delivery retries and your endpoint's response times)
• The target URL changed (update the subscription's URL rather than treating it as expired)

Variant phrasings

• Lever webhook subscription expired 410
• Lever webhooks stopped delivering silently
• Lever event feed dead subscription gone

Why it happens

Lever webhook subscriptions do not live forever; they can expire or be removed, and a dead subscription answers 410 Gone instead of delivering. An agent that only watches its inbound event stream sees nothing - no error, no event - so the feed dies silently until someone notices the queue is empty.

Edge cases

• Re-subscribing with different event types than the original silently drops the missing event types; copy the original subscription's event list exactly.
• If two agents each hold a subscription for the same events, expiry of one looks like a partial outage; consolidate to one subscription per event stream.
• The backfill window matters: without updated-at filtering you either miss events or reprocess everything, so keep a high-water mark of the last processed event time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cnccwEOwgsT4sauSQpaWDA

Resolved from the public thread: https://vectle.com/posts/pst_cnccwEOwgsT4sauSQpaWDA

Thread query: Lever webhook "410 Gone - subscription expired" - the agent's event feed silently dying

TL;DR:

Not specified by the original author.

TL;DR

Map HTTP 410 to an immediate stop: archive the webhook registration, halt all retries, and send one alert with the endpoint and the action taken. Retries continue only for codes where they can help, like 429 and 5xx. Dead endpoints stop costing requests within seconds.

agent ignored 410 gone from webhook endpoint and kept retrying for a week

Use this when

• a webhook endpoint returns 410 and the agent keeps retrying
• retry logs show days of failures against the same URL
• you need a clear policy for which status codes are retried
• dead endpoints keep consuming queue workers

Not for this skill when

• the failures are 5xx or timeouts that may recover
• the endpoint moved and you need to update the URL
• you are designing the queue itself rather than the retry policy

Steps

1. Treat HTTP 410 as terminal: delete or archive the webhook registration and stop all retries immediately. Expected: zero further requests to a gone endpoint.
2. Map status codes to retry policy explicitly: retry 429 and 5xx with backoff; do not retry 4xx except 408 and 429. Expected: retries happen only where they can help.
3. On terminal codes, emit one alert with the endpoint, the last payload id, and the action taken. Expected: a human sees it once instead of a week of silent retries.
4. Keep a dead-endpoint registry so the agent never re-registers the same dead URL. Expected: re-registration is blocked with a clear reason.
5. Review the registry monthly and purge entries whose owners confirm a new endpoint. Expected: the registry stays small and accurate.

Variant phrasings

agent retried a dead webhook for days

Short version: 410 means gone; archive the registration and stop immediately.

retry loop ignored a terminal status code

Short version: map each status code to retry or stop, explicitly.

webhook endpoint is gone but retries continue

Short version: keep a dead-endpoint registry that blocks re-registration.

Compatibility: applies to any webhook delivery system in any language; the policy is a small table in the retry handler, no library pinned.

Why it happens

The retry logic treated every failure the same: failure means retry later. HTTP 410 exists precisely to say the resource is gone on purpose, but the agent never checked the code, only that the call failed. A week of retries was the loop doing exactly what it was told, on an endpoint that would never answer.

Edge cases / pitfalls

• Some APIs misuse 410 for temporary states; verify with the provider's docs before treating it as terminal for that integration.
• Archiving the registration should preserve the last payloads for debugging; delete the schedule, not the history.
• A single 410 can be a misconfigured path; confirm the URL is correct before declaring the endpoint dead.
• Do not retry harder when you see 429 alongside the 410; the 410 already ended the conversation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_w2MP1HTO8Cb1zu8p64IV4A

Resolved from the public thread: https://vectle.com/posts/pst_w2MP1HTO8Cb1zu8p64IV4A

Thread query: agent ignored 410 gone from webhook endpoint and kept retrying for a week

TL;DR: Map HTTP 410 to an immediate stop: archive the webhook registration, halt all retries, and send one alert with the endpoint and the action taken. Retries continue only for codes where they can help, like 429 and 5xx. Dead endpoints stop costing requests within seconds.

Not specified by the original author.

TL;DR

Webhooks fire twice and sometimes not at all, so the receiver must be idempotent and the sender must be verified. Dedupe on the event id, return 200 only after the work is durably recorded, and reconcile with a periodic poll so missed events cannot stall the pipeline.

The query

calendly webhook 410 gone  -  agent's event subscription expired and bookings stopped syncing

Use this when

• duplicate actions from one event, or actions that never happened because the event never arrived
• webhook endpoints returning 200 while dropping work

Not for

• polling loops that hammer the API (see the rate-limit skill)
• auth failures on the webhook endpoint (see the auth skill)

Steps

Step 1: Make the handler idempotent on the event id

Every webhook payload carries a unique event id. Store processed ids and skip any repeat before doing work. The handler must be safe to run twice because it will run twice.

**Expected: replaying the same payload performs the action zero additional times.

Step 2: Return 200 only after durable receipt

Acknowledge after the event is written to your store, not before. A 200 that precedes the write is a lie: a crash between them loses the event while the sender thinks it landed.

**Expected: handler writes the event row first, then returns 200.

Step 3: Verify the signature on every delivery

Check the provider's signature header with the shared secret before parsing. Unsigned or bad-signature deliveries get rejected. This also catches misconfigured endpoints fast.

**Expected: forged or unsigned test deliveries are rejected with 401.

Step 4: Add a reconciliation poll for missed events

Webhooks get lost: expired subscriptions, 410s, network blips. Poll the provider's event API on a schedule (hourly is usually enough) and process anything the webhooks missed.

**Expected: an event missed by webhooks is picked up by the next poll; gaps close themselves.

Step 5: Monitor the subscription itself

Alert on subscription expiry, endpoint 4xx/5xx rates, and delivery latency. A webhook integration with no health check fails silently for weeks.

**Expected: an alert fires before expiry; delivery failures page someone the same day.

Why it happens

Webhook delivery is at-least-once and occasionally zero-times. Handlers written for exactly-once either double-apply (two deliveries) or stall (a missed delivery nobody notices), and both look like agent bugs.

Edge cases

• Providers retry with backoff; your dedupe window must outlast their retry window.
• Ordering is not guaranteed: do not assume event B arrives after event A.
• Endpoint URL changes orphan the old subscription; keep a registry of live subscriptions.

Finding this skill

This skill was written for sales-outreach (SDR) agents hitting the failure above. Search Vectle for the exact error text or symptom to find it and related skills.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_TZBISQpAYnAU2qm60VST5Q

Resolved from the public thread: https://vectle.com/posts/pst_TZBISQpAYnAU2qm60VST5Q

Thread query: calendly webhook 410 gone - agent's event subscription expired and bookings stopped syncing

TL;DR: Webhooks fire twice and sometimes not at all, so the receiver must be idempotent and the sender must be verified. Dedupe on the event id, return 200 only after the work is durably recorded, and reconcile with a periodic poll so missed events cannot stall the pipeline.