dashboard alert did not fire: threshold query failed with error in Grafana

TL;DR

Open the alert rule and fix its query first, because a rule whose query errors or returns no series sits in an error or no data state and never evaluates the threshold. Correct the query, check the rule evaluation history, and run a test evaluation with data that crosses the threshold. Only after the rule evaluates cleanly should the notification path be suspected.

dashboard alert did not fire: threshold query failed with error in Grafana

Use this when

• An incident happened and the expected Grafana alert never notified anyone.
• The alert rule shows an error, no data, or a paused state in the alerting list.
• The underlying metric or recording rule was renamed or stopped recently.

Not for this skill when

• The alert fired and appears in the alert history but nobody was notified. Debug contact points and notification policies instead.
• The panel for the same data also errors for every viewer. Fix the data source or metric pipeline first.

Steps

1. Open the alert rule state history for the incident window and record the state the rule held. Verify: the history shows an error, no data, or normal state instead of a firing state
2. Edit the rule and run its query in the rule preview, correcting any renamed metric, wrong data source, or broken label filter. Verify: the preview returns the expected series for the incident period
3. Check the reduce and threshold settings against the series shape so the comparison uses the intended value. Verify: the preview evaluation crosses the threshold on incident shaped data
4. Review the rule labels and the notification policy matching so the evaluated alert routes to a contact point. Verify: the policy match preview shows the intended receiver
5. Trigger a controlled test, such as a temporary low threshold in a test rule copy, and confirm a notification arrives. Verify: the test alert reaches the contact point and resolves
6. Restore the production threshold, unpause the rule if needed, and confirm its state returns to normal evaluation. Verify: the rule list shows the rule active with a healthy last evaluation

Why it happens

Silent alert failures are worse than loud ones because the system looks monitored while it is not. The state history is the ground truth, it records what the rule did, not what it was configured to do. Periodic test fires and a separate watchdog on alert evaluation health are how teams keep the absence of an alert meaningful.

Grafana alerting evaluates the rule query on a schedule, reduces the result, and compares it with the threshold condition. Each stage can quietly neutralise the rule, a query broken by a renamed metric, a reduce step over an empty result, or labels that make the alert instance differ from what the notification policy expects. The rule state history shows which state the rule actually sat in during the incident window, which is the evidence that separates a query failure from a delivery failure.

Field notes for Grafana specifically. The server log at startup and the provisioning and alerting sections are where instance level truth lives, and the dashboard JSON model view in the interface shows exactly what Grafana stored, which settles most arguments about what a panel is really querying. Grafana changes behaviour across versions in the alerting and provisioning areas especially, so record the version in every incident note. Instances managed as code should treat the Git copy of a dashboard as the source to fix, with interface edits treated as drafts to be ported back.

A Grafana habit that prevents repeat incidents is to keep one small health dashboard for the monitoring system itself, showing data source query success, alert evaluation state, and provisioning status. Monitoring the monitor sounds recursive, but the failures in this family are silent by nature, empty variables, unevaluated rules, missing dashboards, and a health view is the only surface where silence itself becomes visible. Checking it after every upgrade, migration, or provisioning change catches the breakage in the same hour it was introduced.

Variants and edge cases

Grafana alert did not fire query error rule state history

The state history phrasing for proving what the rule did.

Grafana alerting no data state threshold never evaluated

The no data variant where the query returns nothing.

Grafana alert rule metric renamed fix query test fire

The rename variant ending in a controlled test fire.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_43-rHDqPZWxUq4-Mao97qw