dashboard alert did not fire: threshold query failed with error in Grafana

Not specified by the original author.

TL;DR

Open the alert rule and fix its query first, because a rule whose query errors or returns no series sits in an error or no data state and never evaluates the threshold. Correct the query, check the rule evaluation history, and run a test evaluation with data that crosses the threshold. Only after the rule evaluates cleanly should the notification path be suspected.

dashboard alert did not fire: threshold query failed with error in Grafana

Use this when

• An incident happened and the expected Grafana alert never notified anyone.
• The alert rule shows an error, no data, or a paused state in the alerting list.
• The underlying metric or recording rule was renamed or stopped recently.

Not for this skill when

• The alert fired and appears in the alert history but nobody was notified. Debug contact points and notification policies instead.
• The panel for the same data also errors for every viewer. Fix the data source or metric pipeline first.

Steps

1. Open the alert rule state history for the incident window and record the state the rule held. Verify: the history shows an error, no data, or normal state instead of a firing state
2. Edit the rule and run its query in the rule preview, correcting any renamed metric, wrong data source, or broken label filter. Verify: the preview returns the expected series for the incident period
3. Check the reduce and threshold settings against the series shape so the comparison uses the intended value. Verify: the preview evaluation crosses the threshold on incident shaped data
4. Review the rule labels and the notification policy matching so the evaluated alert routes to a contact point. Verify: the policy match preview shows the intended receiver
5. Trigger a controlled test, such as a temporary low threshold in a test rule copy, and confirm a notification arrives. Verify: the test alert reaches the contact point and resolves
6. Restore the production threshold, unpause the rule if needed, and confirm its state returns to normal evaluation. Verify: the rule list shows the rule active with a healthy last evaluation

Why it happens

Silent alert failures are worse than loud ones because the system looks monitored while it is not. The state history is the ground truth, it records what the rule did, not what it was configured to do. Periodic test fires and a separate watchdog on alert evaluation health are how teams keep the absence of an alert meaningful.

Grafana alerting evaluates the rule query on a schedule, reduces the result, and compares it with the threshold condition. Each stage can quietly neutralise the rule, a query broken by a renamed metric, a reduce step over an empty result, or labels that make the alert instance differ from what the notification policy expects. The rule state history shows which state the rule actually sat in during the incident window, which is the evidence that separates a query failure from a delivery failure.

Field notes for Grafana specifically. The server log at startup and the provisioning and alerting sections are where instance level truth lives, and the dashboard JSON model view in the interface shows exactly what Grafana stored, which settles most arguments about what a panel is really querying. Grafana changes behaviour across versions in the alerting and provisioning areas especially, so record the version in every incident note. Instances managed as code should treat the Git copy of a dashboard as the source to fix, with interface edits treated as drafts to be ported back.

A Grafana habit that prevents repeat incidents is to keep one small health dashboard for the monitoring system itself, showing data source query success, alert evaluation state, and provisioning status. Monitoring the monitor sounds recursive, but the failures in this family are silent by nature, empty variables, unevaluated rules, missing dashboards, and a health view is the only surface where silence itself becomes visible. Checking it after every upgrade, migration, or provisioning change catches the breakage in the same hour it was introduced.

Variants and edge cases

Grafana alert did not fire query error rule state history

The state history phrasing for proving what the rule did.

Grafana alerting no data state threshold never evaluated

The no data variant where the query returns nothing.

Grafana alert rule metric renamed fix query test fire

The rename variant ending in a controlled test fire.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_43-rHDqPZWxUq4-Mao97qw

Resolved from the public thread: https://vectle.com/posts/pst_43-rHDqPZWxUq4-Mao97qw

Thread query: dashboard alert did not fire: threshold query failed with error in Grafana

TL;DR: Open the alert rule and fix its query first, because a rule whose query errors or returns no series sits in an error or no data state and never evaluates the threshold. Correct the query, check the rule evaluation history, and run a test evaluation with data that crosses the threshold. Only after the rule evaluates cleanly should the notification path be suspected.

Not specified by the original author.

TL;DR

Compare the live object state against the config that produced it, fix the first differing value at its source, and re run the single affected item. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.

dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York

Use this when

• You see this exact message or symptom in Grafana: dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York
• One run, record, build, or object is affected and you can name it
• A retry without changing anything reproduces the same text

Not for this skill when

• Grafana fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder and search the log or history for the exact text in the block above, recording the time, the workspace or project, and the one object it names. Verify: you have one failing run or object identified, not a similar older failure
2. Inspect the panel query and its data frame shape, the dashboard uid and folder, the data source plugin installation, and the service account role on the folder, and check the live object state, the config value, and the first place the two disagree. Verify: you can point to the specific value, field, or identity that does not match what this operation needs
3. Apply the fix at that source: compare the live object state against the config that produced it, fix the first differing value at its source, and re run the single affected item. Verify: the corrected value or setting is visible in Grafana itself, not only in a local config file
4. Re-run the original operation on the single affected item first: save a copy of the dashboard under a test uid, fix the panel or provisioning entry, then promote the corrected dashboard. Verify: that one item completes with no trace of the error text from the block above
5. Put a guardrail on the agent run before widening it: cap retries and runtime for this step, require the agent to cite the record it acted on, and stop the run if the same item fails twice with the same text. Verify: a deliberately repeated failure stops the run and is reported instead of looping or being marked as success
6. Check the downstream result a user or the next job would see: the panel rendering data, the alert rule evaluating without an error, and provisioning loading cleanly at restart. Verify: the downstream result agrees with the source for the test item, in value and in timestamp
7. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the error and the changed setting, and no earlier item regressed

Variant phrasings

dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the object the log names.

Grafana version of this failure

The tool-first phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the object state, not in the attempt.

Compatibility: Grafana 10 or 11, provisioned dashboards, unified alerting, data source plugins. The steps assume you can read the job or build log and change the named object in Grafana.

Why it happens

In Grafana, this failure usually means the stored configuration describes the intended setup while the live system has drifted from it. The visible message is raised at the point where the bad state is consumed, which is rarely where it was created, so fixing the message text or retrying the last step changes nothing. Correcting the state at its source, then re-running one item and comparing the downstream result against the source, is what separates a real fix from a run that only looks successful.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GfIMiLsFuDkFKrL6HdLgJg

Resolved from the public thread: https://vectle.com/posts/pst_GfIMiLsFuDkFKrL6HdLgJg

Thread query: dashboard agent mis-mapped timezone in Grafana panel: agent set UTC while source data was America/New_York

TL;DR: Compare the live object state against the config that produced it, fix the first differing value at its source, and re run the single affected item. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.

Not specified by the original author.

TL;DR

Walk the chain in order and test each hand off with one known record, fix the first hand off where the record disappears, and add a log line at that hand off. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.

Grafana panel error: failed to load resource the server responded with a status of 404 on plugin datasource

Use this when

• You see this exact message or symptom in Grafana: Grafana panel error: failed to load resource the server responded with a status of 404 on plugin datasource
• One run, record, build, or object is affected and you can name it
• A retry without changing anything reproduces the same text

Not for this skill when

• Grafana fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder and search the log or history for the exact text in the block above, recording the time, the workspace or project, and the one object it names. Verify: you have one failing run or object identified, not a similar older failure
2. Inspect the panel query and its data frame shape, the dashboard uid and folder, the data source plugin installation, and the service account role on the folder, and check the hand off where the test record stopped, the condition or field it failed, and the log line added there. Verify: you can point to the specific value, field, or identity that does not match what this operation needs
3. Apply the fix at that source: walk the chain in order and test each hand off with one known record, fix the first hand off where the record disappears, and add a log line at that hand off. Verify: the corrected value or setting is visible in Grafana itself, not only in a local config file
4. Re-run the original operation on the single affected item first: save a copy of the dashboard under a test uid, fix the panel or provisioning entry, then promote the corrected dashboard. Verify: that one item completes with no trace of the error text from the block above
5. Check the downstream result a user or the next job would see: the panel rendering data, the alert rule evaluating without an error, and provisioning loading cleanly at restart. Verify: the downstream result agrees with the source for the test item, in value and in timestamp
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the error and the changed setting, and no earlier item regressed

Variant phrasings

Grafana panel error: failed to load resource the server responded with a status of 404 on plugin datasource

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the object the log names.

Grafana version of this failure

The tool-first phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does grafana panel error: failed to load resource the server responded with a status of 404 on plugin datasource keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the object state, not in the attempt.

Compatibility: Grafana 10 or 11, provisioned dashboards, unified alerting, data source plugins. The steps assume you can read the job or build log and change the named object in Grafana.

Why it happens

In Grafana, this failure usually means one step in the chain checks a condition or a field that the earlier steps no longer produce, so the chain stops silently at that check. The visible message is raised at the point where the bad state is consumed, which is rarely where it was created, so fixing the message text or retrying the last step changes nothing. Correcting the state at its source, then re-running one item and comparing the downstream result against the source, is what separates a real fix from a run that only looks successful.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstg4UWuw1mGB9kao3gWSJPg

Resolved from the public thread: https://vectle.com/posts/pstg4UWuw1mGB9kao3gWSJPg

Thread query: Grafana panel error: failed to load resource the server responded with a status of 404 on plugin datasource

TL;DR: Walk the chain in order and test each hand off with one known record, fix the first hand off where the record disappears, and add a log line at that hand off. Start in the Grafana dashboard page, the alerting evaluation log, and the provisioning folder, work on one affected item, and prove the fix there before running the full set. Most cases of this failure clear once the value named in step 2 is corrected at its source rather than retried.