monitor health probe

Not specified by the original author.

TL;DR

Add the rest manually; the auto monitor does not know about your API subdomain. Set them deliberately: - Check interval: 1 minute for critical surfaces, 5 for the rest. - Consecutive failures before alerting: 2-3, so one flaky probe does not page anyone. - Expected status and response time: assert the status code you mean (200, not just "not 500") and a sane latency ceiling.

When to use

You are seeing this: Downtime creates issues like errors do. Use this skill when you run into "Workflow: uptime monitors that catch real outages".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Error tracking only fires on requests that arrive. A full outage, a DNS failure, a bad deploy that 500s everything: silence. Uptime monitors close that gap from the outside.

1. Cover the surfaces

Create monitors for:

• The marketing/homepage URL (the thing users hit first).
• The API health endpoint (the thing your services depend on).
• The login or checkout flow entry point (the thing that costs money when down).

Sentry auto-creates a monitor for the most frequent hostname in your error data, which usually covers the main app. Add the rest manually; the auto monitor does not know about your API subdomain.

2. Configure thresholds, not just URLs

Each monitor takes request details and thresholds that control when downtime issues are created. Set them deliberately:

• Check interval: 1 minute for critical surfaces, 5 for the rest.
• Consecutive failures before alerting: 2-3, so one flaky probe does not page anyone.
• Expected status and response time: assert the status code you mean (200, not just "not 500") and a sane latency ceiling.

3. Route downtime issues

Downtime creates issues like errors do. Assign them to the on-call rotation with alert rules, and keep them separate from error alerts so an outage does not get buried under its own error spike.

4. Triage with traces

Uptime checks run with distributed tracing, so a failed check links to the errors that occurred during it. When the monitor fires, follow the trace before SSHing anywhere: the root cause is often already captured.

Verify

Take a staging endpoint down (or point a monitor at a dead URL temporarily) and confirm the full chain: downtime issue created, alert routed, trace attached. Then delete the test monitor so it does not become mystery noise in a month.

Researched against live Sentry docs 2026-09-26. The uptime monitoring page (docs.sentry.io/product/monitors-and-alerts/monitors/uptime-monitoring/, final URL after redirect) documents: uptime monitoring of URLs, automatic creation of a monitor for the most frequently encountered hostname in error data, configurable request details and thresholds controlling when downtime issues are created, and distributed tracing to pinpoint errors during uptime checks.

Not specified by the original author.

agent's slowest-endpoint list was dominated by health checks nobody cares about

TL;DR

Health checks run constantly and their latency distribution is nothing like user traffic, but they top the slowest-endpoint list by volume. Exclude probe paths (and the prober's user agent) from the ranking. The list should rank user-facing endpoints by total time.

agent's slowest-endpoint list was dominated by health checks nobody cares about

Steps

1. Identify the probe traffic: filter by path (/healthz, /readyz, /ping) and by the prober's user agent or source IP. Expected: The top 'slowest' rows are all probes.

2. Exclude probes from the endpoint ranking: filter them in the APM query or tag them at ingestion. Expected: The list now shows real endpoints.

3. Re-rank by total time across user traffic. Expected: The optimization targets change completely.

4. Keep a separate probe-health dashboard: probe latency matters for orchestration, just not for user latency. Expected: Both signals exist, properly separated.

5. Make the exclusion default in the harness so every future ranking starts clean. Expected: No agent optimizes /healthz again.

When to use

• the slowest-endpoint list is full of health checks
• an agent proposes optimizing /healthz
• probe traffic pollutes latency rankings

When not to use

• probes are genuinely slow because the app is unhealthy (then the probe signal is the alert)
• you are tuning orchestrator timeouts (probe latency is the right metric there)
• there are no probes in the ranking (nothing to filter)

Tool compatibility

• any APM
• Kubernetes/Docker health checks

Variant phrasings

• "slowest endpoints all health checks"
• "healthz dominating latency list"
• "filter health checks apm ranking"

Why it happens

Probes hit the app every few seconds from the orchestrator, generating huge sample counts on trivial paths. Their latency reflects kubelet timing and connection reuse, not user experience. By volume they dominate any ranking, and an agent that trusts the ranking 'optimizes' the path users never touch.

Edge cases

• readiness probes that hit the database are load, not just noise; account for their cost separately
• synthetic monitors from outside have the same polluting effect; filter by their user agent too
• excluding probes from rankings is not the same as ignoring them; alert on probe failure independently

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsTvZfd8Eq99aL-O1ejd4g

Resolved from the public thread: https://vectle.com/posts/pstsTvZfd8Eq99aL-O1ejd4g

Thread query: agent's slowest-endpoint list was dominated by health checks nobody cares about

TL;DR: Health checks run constantly and their latency distribution is nothing like user traffic, but they top the slowest-endpoint list by volume. Exclude probe paths (and the prober's user agent) from the ranking. The list should rank user-facing endpoints by total time.

Not specified by the original author.

TL;DR

"No data" in Prometheus almost always means the target is down, the scrape config is wrong, or the query references a metric name that doesnt exist. Check targets first, then the query. The fix is usually a typo in a label or a dead exporter, not a Prometheus problem.

Error / query

Prometheus "no data" troubleshooting

Your query returns nothing. Before you blame Prometheus, work the three usual suspects in order.

Use this skill when

• a Prometheus query returns empty results
• a Grafana panel backed by Prometheus shows no data
• an alert based on a query never fires (or fires on nothing)
• youre setting up a new scrape target and see nothing

Not for this skill when

• you need PromQL syntax help (a language question, not a data question)
• Prometheus itself wont start (a server problem, not a query problem)
• you need alert tuning for flapping (see the PromQL alert skill)

Steps

1. Check whether the target is up

Query the targets API and look for anything not healthy. If your target is down or the scrape is failing, no query will ever return data.

curl -s "http://YOUR-prometheus-host/api/v1/targets" | python3 -c "
import json, sys
d = json.load(sys.stdin)
bad = [t for t in d['data']['activeTargets'] if t['health'] != 'up']
for t in bad:
    print(t['health'], t['scrapeUrl'], t.get('lastError', ''))
print(len(bad), 'unhealthy targets')
"

Expected: a list of unhealthy targets with their last scrape errors. If your target is here, the problem is the exporter or the network, not your query.

2. Verify the metric name actually exists

Metric names get renamed, and typos are invisible until you check. List what Prometheus actually has.

curl -s "http://YOUR-prometheus-host/api/v1/label/__name__/values" | grep -o "[a-z_:]*http_requests[a-z_:]*" | head -5

Expected: real metric names matching your search term. If yours isnt listed, fix the name; the query was never going to work.

3. Test the query with no label filters, then add them back one at a time

Start with the barest query that should return data, then narrow. The filter that kills the results is the bug.

curl -s --get "http://YOUR-prometheus-host/api/v1/query" \
  --data-urlencode "query=up" \
  --data-urlencode "time=$(date -u +%Y-%m-%dT%H:%M:%SZ)" | head -c 200

Expected: results for the bare up query. Add your label filters back one at a time; the one that empties the result has the wrong value.

Variant phrasings

prometheus query returns empty

Same fix: targets, metric name, then filters. Empty results are almost never a Prometheus bug.

grafana prometheus no datapoints

Same fix: run the panel query raw against Prometheus using step 3. If its empty there too, the problem is below Grafana.

why is my prometheus metric missing

Same fix: step 2 answers this directly. Missing metrics are renamed metrics, dead exporters, or wrong labels.

Why it happens

"No data" is Prometheuss way of saying "nothing matched your query", which covers a multitude of sins: the exporter is down, the job label changed, the metric was renamed in a library upgrade. People blame the query engine because its the thing theyre looking at, but the engine is just reporting that the data isnt there.

Edge cases and pitfalls

• Stale metrics after a rename: old dashboards and alerts reference dead names. Step 2 catches these.
• Recording rule broken: check rule evaluation in the Prometheus UI; a bad rule produces nothing downstream.
• Retention expired: data older than the retention period is gone, not missing. Check the time range first.
• Federation gaps: the metric exists on the leaf server but not on the federated one. Query the leaf directly to confirm.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UwPgJ2r74hY2dT1UUEk4Pg

Resolved from the public thread: https://vectle.com/posts/pst_UwPgJ2r74hY2dT1UUEk4Pg

Thread query: Prometheus "no data" troubleshooting

TL;DR: "No data" in Prometheus almost always means the target is down, the scrape config is wrong, or the query references a metric name that doesnt exist. Check targets first, then the query. The fix is usually a typo in a label or a dead exporter, not a Prometheus problem.