agent's slowest-endpoint list was dominated by health checks nobody cares about

agent's slowest-endpoint list was dominated by health checks nobody cares about

TL;DR

Health checks run constantly and their latency distribution is nothing like user traffic, but they top the slowest-endpoint list by volume. Exclude probe paths (and the prober's user agent) from the ranking. The list should rank user-facing endpoints by total time.

agent's slowest-endpoint list was dominated by health checks nobody cares about

Steps

1. Identify the probe traffic: filter by path (/healthz, /readyz, /ping) and by the prober's user agent or source IP. Expected: The top 'slowest' rows are all probes.

2. Exclude probes from the endpoint ranking: filter them in the APM query or tag them at ingestion. Expected: The list now shows real endpoints.

3. Re-rank by total time across user traffic. Expected: The optimization targets change completely.

4. Keep a separate probe-health dashboard: probe latency matters for orchestration, just not for user latency. Expected: Both signals exist, properly separated.

5. Make the exclusion default in the harness so every future ranking starts clean. Expected: No agent optimizes /healthz again.

When to use

• the slowest-endpoint list is full of health checks
• an agent proposes optimizing /healthz
• probe traffic pollutes latency rankings

When not to use

• probes are genuinely slow because the app is unhealthy (then the probe signal is the alert)
• you are tuning orchestrator timeouts (probe latency is the right metric there)
• there are no probes in the ranking (nothing to filter)

Tool compatibility

• any APM
• Kubernetes/Docker health checks

Variant phrasings

• "slowest endpoints all health checks"
• "healthz dominating latency list"
• "filter health checks apm ranking"

Why it happens

Probes hit the app every few seconds from the orchestrator, generating huge sample counts on trivial paths. Their latency reflects kubelet timing and connection reuse, not user experience. By volume they dominate any ranking, and an agent that trusts the ranking 'optimizes' the path users never touch.

Edge cases

• readiness probes that hit the database are load, not just noise; account for their cost separately
• synthetic monitors from outside have the same polluting effect; filter by their user agent too
• excluding probes from rankings is not the same as ignoring them; alert on probe failure independently

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsTvZfd8Eq99aL-O1ejd4g