agent's webhook to ticketing failed and the failure alert was lost

TL;DR

A failure alert sent once over a webhook with no retry is a hope, not a notification. Persist every alert to a local outbox before sending, retry delivery with backoff until acknowledged, escalate to a secondary channel after repeated failure, and reconcile the outbox daily so a lost webhook can delay a ticket but never erase it.

agent's webhook to ticketing failed and the failure alert was lost

Use this when

• A restore test failed but no ticket exists and the team learned from the next run.
• Webhook logs show a timeout or 5xx at the alert moment with no follow-up attempt.
• The ticketing system has no record while the agent log claims it alerted.

Not for this skill when

• The restore test itself failed to run. No alert was owed yet.
• Ticketing creates the ticket but routing sends it to the wrong queue. Delivery worked; routing is the issue.

Steps

1. Inspect the agent alert code path for persistence and retry behaviour. Verify: whether alerts are fire-and-forget is confirmed
2. Implement a local alert outbox written before any network send, with delivery state per alert. Verify: killing the agent mid-alert leaves a recoverable outbox entry
3. Add retry with backoff on webhook failure and mark alerts delivered only on a success response. Verify: a simulated webhook outage shows retries and eventual delivery
4. Add a fallback channel, such as email or a second webhook, after a defined number of failed attempts. Verify: the fallback fires in a test
5. Reconcile outbox versus ticketing records for the past runs and create any missing tickets for real failures. Verify: historical lost alerts are recovered and filed

Variant phrasings

webhook alert lost retry outbox pattern

The outbox phrasing.

agent failure notification failed no ticket

The missing-ticket variant.

dead letter queue alerting agent webhook

The dead-letter wording.

Compatibility: Agents alerting into Jira, ServiceNow, PagerDuty and generic webhook receivers, with local persistent queues in the agent runtime.

Why it happens

Webhooks fail for mundane reasons: receiver deploys, network blips, expired endpoint credentials. A single-attempt design makes alert reliability equal to the worst moment of the receiver, and it fails silently precisely when the news is bad. The outbox pattern, borrowed from messaging systems, is the standard cure because it decouples recording the failure from delivering it.

Edge cases / pitfalls

• Test the alert path on a schedule by injecting a harmless failure. Silent alerting rot is common.
• Include run identifiers in alerts so duplicate deliveries after retries can be deduplicated by the receiver.
• An outbox that grows forever is its own incident. Alert on outbox age as well.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fJi5M3iFfkx73PA1j-QkrQ