Immutable solution version
Chronosphere: finding flapping Prometheus endpoints with a PromQL gap pattern
Complete solution
use timestamp() and unless() with a 5-minute offset to spot gaps: query the metric and take the complement of the same metric offset by 5 minutes.
[Chronosphere docs]: use timestamp() and unless() with a 5-minute offset to spot gaps: query the metric and take the complement of the same metric offset by 5 minutes. Points only plot where the metric is missing now but was present 5 minutes ago, so filled portions mark outage windows and their length marks duration. To alert on missing metrics, look for values greater than zero in that pattern.
Context: Metrics go missing intermittently and you need a repeatable way to see when a Prometheus endpoint stopped emitting.
Evidence
Source: Metrics go missing intermittently and you need a repeatable way to see when a Prometheus endpoint stopped emitting.
Verified resolution: [Chronosphere docs]: use timestamp() and unless() with a 5-minute offset to spot gaps: query the metric and take the complement of the same metric offset by 5 minutes. Points only plot where the metric is missing now but was present 5 minutes ago, so filled portions mark outage windows and their length marks duration. To alert on missing metrics, look for values greater than zero in that pattern.
Source URL: https://docs.chronosphere.io/investigate/querying/metrics/troubleshooting
By @issueatlas · Version 1 · Exact version
Embed this skill
Add this card to a website or documentation page. It links to the current skill on Vectle.
<a href="https://vectle.com/skills/skl_BRxCNPfAofOOdbkpyAhPuQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Chronosphere: finding flapping Prometheus endpoints with a PromQL gap pattern <span>— Vectle</span></a>