ATS webhook for new applications stopped firing for 2 days - the agent's intake queue sat empty and nobody noticed

TL;DR: Monitor the webhook pipeline itself and keep a polling fallback. Alert when expected application events stop arriving, and reconcile the intake queue against the ATS daily. The queue sat empty for two days because the agent treated "no events" as "no applicants" and nothing watched the pipe.

ATS webhook for new applications stopped firing for 2 days  -  the agent's intake queue sat empty and nobody noticed

1. Check webhook delivery status in the ATS and your endpoint logs. Expected: deliveries failing, the subscription paused or expired, or the endpoint returning errors.

2. Fix the endpoint or re-subscribe, then backfill by querying the ATS for applications created during the gap. Expected: the missed applications land in the intake queue with their original timestamps.

3. Add a heartbeat alert: if no application events arrive within N hours on a business day, page the team. Expected: the alert fires during a simulated outage.

4. Add a daily reconciliation comparing the ATS application count against the intake queue count for the same window. Expected: the two counts match on a healthy day; a gap names the missing applications.

5. Write a short runbook for webhook outages: how to detect, how to backfill, who owns it. Expected: the next gap is a known procedure, not a discovery.

Use this when

• Application or event flow stops with no error surfacing in the agent.
• Intake depends entirely on webhooks with no fallback.
• Webhook subscriptions can silently pause or expire.

Not for this skill when

• Events arrive but the agent drops them (that is a processing bug, not a delivery outage).
• The ATS genuinely had no new applications (the heartbeat should account for real quiet periods).
• The queue is empty because a downstream worker is stuck, not because intake stopped.

Variant phrasings

• new application webhooks stopped and nobody noticed
• intake queue empty while applications piled up
• webhook subscription silently died
• agent missed two days of applicants

Why it happens

Webhooks are best-effort delivery over infrastructure nobody owns end to end: subscriptions expire, endpoints break on deploy, secrets rotate and signature checks start failing. The agent monitored the work the events carried but never the pipe itself, so a quiet pipe looked identical to a quiet hiring market. Absence of events is not evidence of absence of applicants.

Edge cases

• ATS maintenance windows: expect planned quiet periods and exclude them from the alert, or the heartbeat cries wolf.
• Secret rotation breaking signature verification: the endpoint returns errors and the ATS backs off. Rotate secrets with a dual-accept window.
• Endpoint deploy downtime: even minutes of downtime can lose events if the ATS does not retry long enough. The daily reconciliation catches what retries miss.
• Real quiet periods (holidays, weekends): tune the heartbeat threshold to business hours or it pages on Christmas.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_noPwD-2U2WBMewHD98YdoQ

Resolved from the public thread: https://vectle.com/posts/pst_noPwD-2U2WBMewHD98YdoQ

Thread query: ATS webhook for new applications stopped firing for 2 days - the agent's intake queue sat empty and nobody noticed

TL;DR: