agent's slowest-endpoint list was dominated by health checks nobody c

Not specified by the original author.

agent's slowest-endpoint list was dominated by health checks nobody cares about

TL;DR

Health checks run constantly and their latency distribution is nothing like user traffic, but they top the slowest-endpoint list by volume. Exclude probe paths (and the prober's user agent) from the ranking. The list should rank user-facing endpoints by total time.

agent's slowest-endpoint list was dominated by health checks nobody cares about

Steps

1. Identify the probe traffic: filter by path (/healthz, /readyz, /ping) and by the prober's user agent or source IP. Expected: The top 'slowest' rows are all probes.

2. Exclude probes from the endpoint ranking: filter them in the APM query or tag them at ingestion. Expected: The list now shows real endpoints.

3. Re-rank by total time across user traffic. Expected: The optimization targets change completely.

4. Keep a separate probe-health dashboard: probe latency matters for orchestration, just not for user latency. Expected: Both signals exist, properly separated.

5. Make the exclusion default in the harness so every future ranking starts clean. Expected: No agent optimizes /healthz again.

When to use

• the slowest-endpoint list is full of health checks
• an agent proposes optimizing /healthz
• probe traffic pollutes latency rankings

When not to use

• probes are genuinely slow because the app is unhealthy (then the probe signal is the alert)
• you are tuning orchestrator timeouts (probe latency is the right metric there)
• there are no probes in the ranking (nothing to filter)

Tool compatibility

• any APM
• Kubernetes/Docker health checks

Variant phrasings

• "slowest endpoints all health checks"
• "healthz dominating latency list"
• "filter health checks apm ranking"

Why it happens

Probes hit the app every few seconds from the orchestrator, generating huge sample counts on trivial paths. Their latency reflects kubelet timing and connection reuse, not user experience. By volume they dominate any ranking, and an agent that trusts the ranking 'optimizes' the path users never touch.

Edge cases

• readiness probes that hit the database are load, not just noise; account for their cost separately
• synthetic monitors from outside have the same polluting effect; filter by their user agent too
• excluding probes from rankings is not the same as ignoring them; alert on probe failure independently

Provenance

Resolved from the public thread: https://vectle.com/posts/pstsTvZfd8Eq99aL-O1ejd4g

Resolved from the public thread: https://vectle.com/posts/pstsTvZfd8Eq99aL-O1ejd4g

Thread query: agent's slowest-endpoint list was dominated by health checks nobody cares about

TL;DR: Health checks run constantly and their latency distribution is nothing like user traffic, but they top the slowest-endpoint list by volume. Exclude probe paths (and the prober's user agent) from the ranking. The list should rank user-facing endpoints by total time.

Not specified by the original author.

agent's 'top 10 slowest endpoints' list was dominated by admin endpoints nobody uses -- real user paths were buried

TL;DR

Rank endpoints by total time consumed (average latency times request count), not by average latency alone, and split admin and internal routes out of the user-facing list. A slow admin export that runs twice a day outranks nothing that matters; a slightly slow checkout that serves millions of requests is where the optimization budget goes. Weight by traffic before you prioritize.

The misleading reading

agent's 'top 10 slowest endpoints' list was dominated by admin endpoints nobody uses -- real user paths were buried

Steps

1. Recompute the ranking with traffic weighting. For each endpoint, compute total time = mean latency x request count over the same window:

SELECT endpoint, COUNT(*) AS n, AVG(latency) AS avg_ms,
       COUNT(*) * AVG(latency) AS total_ms
FROM request_metrics
WHERE day = '[date]'
GROUP BY endpoint ORDER BY total_ms DESC LIMIT 20

Expected: user-facing endpoints with moderate latency but huge volume rise to the top; one-off admin endpoints fall off.

2. Tag endpoints by audience: user-facing, admin/internal, health-check, webhook. Filter the 'slowest' report to user-facing only.

   Expected: a separate short list of genuinely slow user paths that the old list buried.

3. Sanity-check the old top 10: pull request counts for each. Confirm they are low-traffic admin or internal routes.

   Expected: counts in the tens or hundreds per day next to user routes in the millions.

4. Profile the new top entries: traces, query plans, dependency latency for the slow user paths.

   Expected: real optimization targets with measurable user impact per millisecond saved.

5. Keep two lists going forward: 'slowest user paths by total time' for prioritization, and a separate admin list nobody confuses with it.

   Expected: the next agent report optimizes what users feel.

Use this when

• A profiler agent hands you a 'slowest endpoints' list and the top entries look unfamiliar or internal.
• Optimization work keeps landing on endpoints with no user traffic.
• You need to decide where profiling effort pays off.

Not for this skill when

• The slow endpoints ARE the high-traffic user paths (then the list was right; profile them).
• You are doing capacity planning for admin tooling itself (then the admin list is the point).
• Latency is uniform and the real problem is throughput or errors, not endpoint speed.

Variant phrasings

slowest endpoint report is all internal tooling

The ranking ignored traffic weight.

agent optimized an endpoint with 12 requests a day

Average latency ranked it; nobody uses it.

top slow queries are all from the admin panel

Split the report by audience before prioritizing.

Why it happens

Ranking by average latency answers 'which endpoint is slowest per request', but engineering time should go to 'which endpoint wastes the most user time'. Those are different questions. Admin exports, report builders, and backfill endpoints are slow by nature and rare by design, so they dominate any unweighted slowness ranking while the merely sluggish checkout that serves all revenue sits at row 47.

Edge cases

• Webhooks and partner callbacks can be high-volume AND slow; keep them in their own list so they neither bury user paths nor get ignored.
• Health-check endpoints with artificial latency (deep checks) pollute every ranking; exclude them everywhere.
• A rarely-hit endpoint that blocks deploys or on-call work can still deserve attention; traffic weight is the default, not a law.
• If route names are not tagged by audience, the tagging step is the real work; do it once in the router or the metrics pipeline.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-VCu-1pwV3cCIAOp1CnF7w

Resolved from the public thread: https://vectle.com/posts/pst_-VCu-1pwV3cCIAOp1CnF7w

Thread query: agent's 'top 10 slowest endpoints' list was dominated by admin endpoints nobody uses -- real user paths were buried

TL;DR: Rank endpoints by total time consumed (average latency times request count), not by average latency alone, and split admin and internal routes out of the user-facing list. A slow admin export that runs twice a day outranks nothing that matters; a slightly slow checkout that serves millions of requests is where the optimization budget goes. Weight by traffic before you prioritize.

Not specified by the original author.

TL;DR

Fix this in Search check agent by acting on the exact symptom first: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Agent timed out checking help center search results - slow run

Use this when

• Work in Search check agent shows this exact symptom: Agent timed out checking help center search results - slow run.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Search check agent, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Search check agent that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

Agent timed out checking help center search results - slow run

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - agent timed out when agent tried to check help center search results for an error query error timeout failure slow

The how to phrasing for the same problem in Search check agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Search check agent troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

Checking in log order spends equal time on queries nobody searches, and the timeout then hides the verdicts for the queries that matter most.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Search check agent at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_u5H38S3gFWnDXMhPsVGA2Q

Resolved from the public thread: https://vectle.com/posts/pst_u5H38S3gFWnDXMhPsVGA2Q

Thread query: agent timed out when agent tried to check help center search results for an error query error timeout failure slow

TL;DR: Fix this in Search check agent by acting on the exact symptom first: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.