agent escalation path failed: on-call page not sent, retry routing

TL;DR

An escalation path fails silently when the page request errors and nothing watches the watcher. Make every page request confirmed: check the pager API response, require an acknowledgment within minutes, and escalate to the next person automatically on ack timeout. Add a dead-man alert so a fully failed path pages someone out-of-band. Escalation must be confirmed delivery, not fire-and-forget.

The query

agent escalation path failed: on-call page not sent, retry routing

Use this when

• Escalations fail without anyone noticing
• On-call pages never arrive
• You are designing escalation routing
• Auditing escalation reliability

Not for

• Writing the escalation policy (who gets paged when)
• On-call schedule management
• Running the incident response itself
• Pager vendor selection

Steps

1. Confirm the page request at the API level

Log the pager API request and response for every escalation. A 200 with a page id is sent; anything else is not. Many "page not sent" incidents are 4xx errors nobody logged.

Expected output: every escalation with a logged page id or error.

2. Require acknowledgment with a timeout

A sent page is not a received page. Require the on-call to acknowledge within N minutes. On timeout, escalate to the next person in the chain automatically. The chain must be ordered and complete.

Expected output: ack timeouts wired to automatic next-person escalation.

3. Retry the page send on transient failures

If the pager API errors transiently, retry with backoff before giving up on that person. Distinguish transient (retry) from permanent (move to next person): 5xx and timeouts retry, 4xx moves on.

Expected output: retry policy distinguishing transient from permanent failures.

4. Build the escalation chain with no single point of failure

The chain needs depth: primary, secondary, manager, and finally a broadcast. A chain of one is a single point of failure. Test the full chain end to end quarterly.

Expected output: a multi-level chain, tested end to end.

5. Add a dead-man alert on the escalation path itself

If no acknowledgment arrives from anyone in the chain within the total window, fire an out-of-band alert (different channel, different system). This catches the case where the whole paging path is down.

Expected output: a dead-man alert independent of the paging system.

Template: the checklist

Escalation path reliability:
[ ] Every page request logged with response or error
[ ] Ack timeout with automatic next-person escalation
[ ] Retry policy: transient retries, permanent moves on
[ ] Multi-level chain tested end to end
[ ] Dead-man alert outside the paging system

Variant phrasings

on-call page never arrived

Steps 1 and 2. Confirm the send, require the ack.

escalation silently failed

Steps 1 and 5. Log everything, dead-man the path.

pagerduty escalation not working

Full sequence as the reliability audit.

Why it happens

Escalation paths are rarely-exercised code paths spanning multiple systems: the agent, the pager API, the on-call's phone. Each handoff can fail, and failures are silent because the path is only tested during real incidents. Reliability has to be built in with confirmation at every handoff, because hope is not a delivery guarantee.

Edge cases

• On-call overrides and swaps: the chain must read the live schedule, not a cached one. Stale schedules page the wrong person.
• Phone-side failures: do-not-disturb, dead battery, no signal. Ack timeouts exist for exactly this.
• Test pages versus real pages: mark tests clearly so responders do not learn to ignore pages.
• Regional pager outages: the dead-man alert must use a different provider or channel than the primary path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tJXsM1obC2MJZryr9BIaqA