Prometheus "no data" troubleshooting

TL;DR

"No data" in Prometheus almost always means the target is down, the scrape config is wrong, or the query references a metric name that doesnt exist. Check targets first, then the query. The fix is usually a typo in a label or a dead exporter, not a Prometheus problem.

Error / query

Prometheus "no data" troubleshooting

Your query returns nothing. Before you blame Prometheus, work the three usual suspects in order.

Use this skill when

• a Prometheus query returns empty results
• a Grafana panel backed by Prometheus shows no data
• an alert based on a query never fires (or fires on nothing)
• youre setting up a new scrape target and see nothing

Not for this skill when

• you need PromQL syntax help (a language question, not a data question)
• Prometheus itself wont start (a server problem, not a query problem)
• you need alert tuning for flapping (see the PromQL alert skill)

Steps

1. Check whether the target is up

Query the targets API and look for anything not healthy. If your target is down or the scrape is failing, no query will ever return data.

curl -s "http://YOUR-prometheus-host/api/v1/targets" | python3 -c "
import json, sys
d = json.load(sys.stdin)
bad = [t for t in d['data']['activeTargets'] if t['health'] != 'up']
for t in bad:
    print(t['health'], t['scrapeUrl'], t.get('lastError', ''))
print(len(bad), 'unhealthy targets')
"

Expected: a list of unhealthy targets with their last scrape errors. If your target is here, the problem is the exporter or the network, not your query.

2. Verify the metric name actually exists

Metric names get renamed, and typos are invisible until you check. List what Prometheus actually has.

curl -s "http://YOUR-prometheus-host/api/v1/label/__name__/values" | grep -o "[a-z_:]*http_requests[a-z_:]*" | head -5

Expected: real metric names matching your search term. If yours isnt listed, fix the name; the query was never going to work.

3. Test the query with no label filters, then add them back one at a time

Start with the barest query that should return data, then narrow. The filter that kills the results is the bug.

curl -s --get "http://YOUR-prometheus-host/api/v1/query" \
  --data-urlencode "query=up" \
  --data-urlencode "time=$(date -u +%Y-%m-%dT%H:%M:%SZ)" | head -c 200

Expected: results for the bare up query. Add your label filters back one at a time; the one that empties the result has the wrong value.

Variant phrasings

prometheus query returns empty

Same fix: targets, metric name, then filters. Empty results are almost never a Prometheus bug.

grafana prometheus no datapoints

Same fix: run the panel query raw against Prometheus using step 3. If its empty there too, the problem is below Grafana.

why is my prometheus metric missing

Same fix: step 2 answers this directly. Missing metrics are renamed metrics, dead exporters, or wrong labels.

Why it happens

"No data" is Prometheuss way of saying "nothing matched your query", which covers a multitude of sins: the exporter is down, the job label changed, the metric was renamed in a library upgrade. People blame the query engine because its the thing theyre looking at, but the engine is just reporting that the data isnt there.

Edge cases and pitfalls

• Stale metrics after a rename: old dashboards and alerts reference dead names. Step 2 catches these.
• Recording rule broken: check rule evaluation in the Prometheus UI; a bad rule produces nothing downstream.
• Retention expired: data older than the retention period is gone, not missing. Check the time range first.
• Federation gaps: the metric exists on the leaf server but not on the federated one. Query the leaf directly to confirm.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UwPgJ2r74hY2dT1UUEk4Pg