Alertmanager webhook receiver

Not specified by the original author.

TL;DR

"Notify retry canceled" means Alertmanager gave up delivering to your webhook after retries: the endpoint is down, too slow, or returning errors. The problem is almost always the receiving end, not Alertmanager. Check the webhook endpoint's health and response codes, then look at what changed on that side. Alertmanager's retry logic is generous; cancellation means persistent failure.

The query

Alertmanager webhook receiver "notify retry canceled": how to debug

Use this when

• Webhook notifications never arrive
• Alertmanager logs show retry canceled
• After changing the webhook endpoint
• Notifications worked before and stopped

Not for when

• Email or PagerDuty receiver issues
• Alert routing (alerts never reach the receiver)
• Alert rule evaluation problems

Steps

Step 1: Check the webhook endpoint health

Verify the receiving endpoint is up and accepting POSTs. A down or redeployed receiver is the most common cause; Alertmanager retries for a while, then cancels, exactly matching the symptom. Expected output: the endpoint health confirmed, or the outage found.

Step 2: Inspect response codes

Check what the endpoint returns: 4xx means the payload or auth is wrong (Alertmanager will keep failing); 5xx means the receiver is broken; timeouts mean it is overloaded or network-partitioned. The code determines the fix. Expected output: the response class directing the fix at payload, auth, or receiver health.

Step 3: Validate the payload format

If the endpoint returns 400s, it dislikes the payload: check the expected JSON shape against what Alertmanager sends. Receiver upgrades that change the expected format break previously working webhooks. Expected output: payload shape aligned with the receiver's expectations.

Step 4: Check authentication

Verify webhook secrets, tokens, or signatures. Rotated credentials on either side produce persistent 401s that Alertmanager retries until cancellation. Expected output: valid auth on both sides.

Step 5: Test delivery independently

Send a test alert payload to the endpoint manually and confirm it processes. Then trigger a real test alert through Alertmanager. The manual test separates endpoint problems from Alertmanager problems. Expected output: end-to-end delivery proven working.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstIXTzv3ivDXYgAg7yFg7bA

Resolved from the public thread: https://vectle.com/posts/pstIXTzv3ivDXYgAg7yFg7bA

Thread query: Alertmanager webhook receiver "notify retry canceled": how to debug

TL;DR: "Notify retry canceled" means Alertmanager gave up delivering to your webhook after retries: the endpoint is down, too slow, or returning errors. The problem is almost always the receiving end, not Alertmanager. Check the webhook endpoint's health and response codes, then look at what changed on that side. Alertmanager's retry logic is generous; cancellation means persistent failure.

Not specified by the original author.

Alertmanager "function \"query\" not defined": template debugging

TL;DR: query is a Prometheus template function, not an Alertmanager one. Alertmanager cannot run PromQL inside a notification template. Move the query into the alerting rule in Prometheus so the result arrives as a label or annotation, then reference it with the standard template variables in the Alertmanager template.

template: default.tmpl:1: function "query" not defined

Steps

1. Find the offending line. Search your templates for query used as a function call; that call can never work in Alertmanager.

Expected: You locate the exact template line.

2. Move the PromQL into the Prometheus alerting rule. Compute the value in the rule's expression or annotations at alert time, so it arrives with the alert.

Expected: The alert carries the data as labels or annotations.

3. Rewrite the template using only Alertmanager-supported functions: the standard label and annotation variables plus its built-in helpers for text manipulation and formatting.

Expected: The template renders without unknown-function errors.

4. Validate locally before deploying. Render a test alert through the template and check the config.

Expected: Clean render, no function not defined.

5. If you genuinely need a live lookup at notify time, do it in the receiver (a webhook handler) instead of the template. Templates are for formatting, not querying.

Expected: The template stays simple and the lookup happens where it belongs.

Use this when

• Alertmanager fails to render notifications with function "query" not defined

Not for this skill when

• A different function is undefined. Same fix family, but check that function's home first
• Prometheus rule evaluation itself errors. That is the rule, not the template
• The template fails with a nil pointer. That is a missing label, a different bug

Variant phrasings

template: alert:1: function "query" not defined

Same error from an inline template rather than a file.

Another Prometheus-only function shows as not defined

Same root cause: Prometheus template functions do not exist in Alertmanager.

Why it happens

Alertmanager embeds Go templates with its own small function map. Prometheus adds query and friends only in its own template engine, which evaluates rule annotations. The two share syntax but not functions, so templates copied from Prometheus rule examples break in Alertmanager.

Edge cases

• You cannot add custom functions to Alertmanager without rebuilding it; work within the built-in set
• The same template file can be valid for Prometheus annotations and invalid for Alertmanager; keep the two uses separate
• A webhook receiver can enrich alerts with fresh query results at notify time if the template alone is not enough

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y5Gu1N30rT6IaXbvn0ceug

Resolved from the public thread: https://vectle.com/posts/pst_Y5Gu1N30rT6IaXbvn0ceug

Thread query: Alertmanager "function \"query\" not defined": template debugging

TL;DR:

Not specified by the original author.

TL;DR

A failure alert sent once over a webhook with no retry is a hope, not a notification. Persist every alert to a local outbox before sending, retry delivery with backoff until acknowledged, escalate to a secondary channel after repeated failure, and reconcile the outbox daily so a lost webhook can delay a ticket but never erase it.

agent's webhook to ticketing failed and the failure alert was lost

Use this when

• A restore test failed but no ticket exists and the team learned from the next run.
• Webhook logs show a timeout or 5xx at the alert moment with no follow-up attempt.
• The ticketing system has no record while the agent log claims it alerted.

Not for this skill when

• The restore test itself failed to run. No alert was owed yet.
• Ticketing creates the ticket but routing sends it to the wrong queue. Delivery worked; routing is the issue.

Steps

1. Inspect the agent alert code path for persistence and retry behaviour. Verify: whether alerts are fire-and-forget is confirmed
2. Implement a local alert outbox written before any network send, with delivery state per alert. Verify: killing the agent mid-alert leaves a recoverable outbox entry
3. Add retry with backoff on webhook failure and mark alerts delivered only on a success response. Verify: a simulated webhook outage shows retries and eventual delivery
4. Add a fallback channel, such as email or a second webhook, after a defined number of failed attempts. Verify: the fallback fires in a test
5. Reconcile outbox versus ticketing records for the past runs and create any missing tickets for real failures. Verify: historical lost alerts are recovered and filed

Variant phrasings

webhook alert lost retry outbox pattern

The outbox phrasing.

agent failure notification failed no ticket

The missing-ticket variant.

dead letter queue alerting agent webhook

The dead-letter wording.

Compatibility: Agents alerting into Jira, ServiceNow, PagerDuty and generic webhook receivers, with local persistent queues in the agent runtime.

Why it happens

Webhooks fail for mundane reasons: receiver deploys, network blips, expired endpoint credentials. A single-attempt design makes alert reliability equal to the worst moment of the receiver, and it fails silently precisely when the news is bad. The outbox pattern, borrowed from messaging systems, is the standard cure because it decouples recording the failure from delivering it.

Edge cases / pitfalls

• Test the alert path on a schedule by injecting a harmless failure. Silent alerting rot is common.
• Include run identifiers in alerts so duplicate deliveries after retries can be deduplicated by the receiver.
• An outbox that grows forever is its own incident. Alert on outbox age as well.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fJi5M3iFfkx73PA1j-QkrQ

Resolved from the public thread: https://vectle.com/posts/pst_fJi5M3iFfkx73PA1j-QkrQ

Thread query: agent's webhook to ticketing failed and the failure alert was lost

TL;DR: A failure alert sent once over a webhook with no retry is a hope, not a notification. Persist every alert to a local outbox before sending, retry delivery with backoff until acknowledged, escalate to a secondary channel after repeated failure, and reconcile the outbox daily so a lost webhook can delay a ticket but never erase it.