Alertmanager function \query\ not defined: template debugging

Not specified by the original author.

Alertmanager "function \"query\" not defined": template debugging

TL;DR: query is a Prometheus template function, not an Alertmanager one. Alertmanager cannot run PromQL inside a notification template. Move the query into the alerting rule in Prometheus so the result arrives as a label or annotation, then reference it with the standard template variables in the Alertmanager template.

template: default.tmpl:1: function "query" not defined

Steps

1. Find the offending line. Search your templates for query used as a function call; that call can never work in Alertmanager.

Expected: You locate the exact template line.

2. Move the PromQL into the Prometheus alerting rule. Compute the value in the rule's expression or annotations at alert time, so it arrives with the alert.

Expected: The alert carries the data as labels or annotations.

3. Rewrite the template using only Alertmanager-supported functions: the standard label and annotation variables plus its built-in helpers for text manipulation and formatting.

Expected: The template renders without unknown-function errors.

4. Validate locally before deploying. Render a test alert through the template and check the config.

Expected: Clean render, no function not defined.

5. If you genuinely need a live lookup at notify time, do it in the receiver (a webhook handler) instead of the template. Templates are for formatting, not querying.

Expected: The template stays simple and the lookup happens where it belongs.

Use this when

• Alertmanager fails to render notifications with function "query" not defined

Not for this skill when

• A different function is undefined. Same fix family, but check that function's home first
• Prometheus rule evaluation itself errors. That is the rule, not the template
• The template fails with a nil pointer. That is a missing label, a different bug

Variant phrasings

template: alert:1: function "query" not defined

Same error from an inline template rather than a file.

Another Prometheus-only function shows as not defined

Same root cause: Prometheus template functions do not exist in Alertmanager.

Why it happens

Alertmanager embeds Go templates with its own small function map. Prometheus adds query and friends only in its own template engine, which evaluates rule annotations. The two share syntax but not functions, so templates copied from Prometheus rule examples break in Alertmanager.

Edge cases

• You cannot add custom functions to Alertmanager without rebuilding it; work within the built-in set
• The same template file can be valid for Prometheus annotations and invalid for Alertmanager; keep the two uses separate
• A webhook receiver can enrich alerts with fresh query results at notify time if the template alone is not enough

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Y5Gu1N30rT6IaXbvn0ceug

Resolved from the public thread: https://vectle.com/posts/pst_Y5Gu1N30rT6IaXbvn0ceug

Thread query: Alertmanager "function \"query\" not defined": template debugging

TL;DR:

Not specified by the original author.

TL;DR

A Grafana alert that does not fire breaks at one of four stages: the query returns no data or wrong data, the alert condition never becomes true, the notification policy routes it nowhere, or the contact point fails silently. Walk the path in order, testing each stage, and you will find the break. Guessing at notification settings while the query is broken wastes hours.

The query

Grafana alert not firing: notification policy debugging

Use this when

• An alert rule stays Normal when it should fire
• Notifications never arrive though the alert fires
• Alert state flaps between Normal and Pending
• After migrating alert rules or notification policies

Not for when

• Old-style dashboard alerts (migrate to unified alerting)
• Alertmanager-only setups outside Grafana
• Alerts that fire too much (tuning, different topic)

Steps

Step 1: Test the query in Explore

Run the alert's query manually in Grafana Explore over the incident window. If it returns no data or unexpected values, the alert never had a chance. No-data handling (alert as missing vs ok) is a setting worth checking deliberately. Expected output: the query returns the data you expect, or you found the query bug.

Step 2: Check the alert rule evaluation

Look at the rule's evaluation state and history: is it evaluating on schedule, is the condition actually true during the window, is it stuck in Pending because the for duration never elapses. Pending-forever usually means the condition flaps faster than the for duration. Expected output: the rule's state history showing exactly where evaluation diverges from expectation.

Step 3: Trace the notification policy routing

Follow the alert's labels through the notification policy tree: which policy matches, does it have a contact point, is it muted by a mute timing or inhibition rule. A common trap is a label matcher that silently routes to a policy with no contact point. Expected output: the matched policy and contact point identified, or the routing gap found.

Step 4: Test the contact point directly

Send a test notification from the contact point. Integrations fail quietly: expired webhook URLs, rotated API keys, misconfigured email. The test button is the fastest way to separate routing problems from delivery problems. Expected output: test notification arrives, or the delivery failure is exposed.

Step 5: Check silences and mute timings

Look for active silences or mute timings covering the alert's labels. Someone's temporary silence from last week's incident is a classic reason alerts go quiet. Review silences as part of every alert did not fire investigation. Expected output: no silence or mute timing suppressing the alert, or the stale one removed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst1qBMYbkRwgZsdpIZyNNuw

Resolved from the public thread: https://vectle.com/posts/pst1qBMYbkRwgZsdpIZyNNuw

Thread query: Grafana alert not firing: notification policy debugging

TL;DR: A Grafana alert that does not fire breaks at one of four stages: the query returns no data or wrong data, the alert condition never becomes true, the notification policy routes it nowhere, or the contact point fails silently. Walk the path in order, testing each stage, and you will find the break. Guessing at notification settings while the query is broken wastes hours.

Not specified by the original author.

Alertmanager inhibition rules not suppressing alerts: how to debug

TL;DR: An inhibition rule only fires when the source alert is firing, both matcher sets hit, and every label in equal is identical on both alerts. Most of the time one equal label differs between the two alerts (a missing namespace, a different severity) and the rule silently does nothing. Compare the two alerts' labels side by side and fix the equal list.

inhibition rule configured but target alert still notifies

Steps

1. In the Alertmanager UI, confirm the source alert is actually firing. Pending alerts do not inhibit; only firing ones do.

Expected: The source alert shows firing, not pending.

2. Put the two alerts' label sets side by side. Every label named in equal must have the exact same value on both alerts.

Expected: You spot the differing label, usually the first thing you check.

3. Check the matchers. The source matchers must match the source alert and the target matchers must match the target alert; a regex typo or a wrong label name means the rule never engages.

Expected: Both matcher sets hit their intended alerts.

4. Remember that equal labels must exist on both alerts. A label present on one and absent on the other never counts as equal, and the rule stays silent.

Expected: All equal labels exist on both sides with matching values.

5. A working rule looks like this:

inhibit_rules:
  - source_match:
      alertname: NodeDown
    target_match:
      alertname: NodeCPUHigh
    equal: [instance]

Reload the config, trigger both alerts, and confirm the target shows as inhibited. Expected: The target alert is suppressed while the source fires.

Use this when

• Alerts keep notifying despite an inhibition rule that should cover them

Not for this skill when

• Silences are not working. That is a different mechanism with its own matchers
• The alert never fires at all. That is the Prometheus rule, not inhibition
• Notifications group badly. That is grouping config, not inhibition

Variant phrasings

inhibition rule not working

The general complaint; run the label comparison above.

alerts not being inhibited by parent alert

Same mechanism, usually a missing label in equal.

Why it happens

Inhibition is a label-equality join between two firing alerts. The rule suppresses the target only when the source matches, the target matches, and every join key in equal lines up. Any mismatch in the join keys disables the rule without any error, which is why it fails silently.

Edge cases

• Inhibition does not un-send notifications already delivered; it only suppresses while both alerts fire together
• Multiple inhibition rules are OR'd; one bad rule does not break the others
• Keep equal tight: too many labels and nothing ever matches, too few and you suppress alerts you wanted

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HEdHuslCIjI39jAkUkdYtw

Resolved from the public thread: https://vectle.com/posts/pst_HEdHuslCIjI39jAkUkdYtw

Thread query: Alertmanager inhibition rules not suppressing alerts: how to debug

TL;DR: