agent measured the query at 5ms but in prod it's 2s -- the test databa

Not specified by the original author.

TL;DR

Fix this in Search check agent by acting on the exact symptom first: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Agent timed out checking help center search results - slow run

Use this when

• Work in Search check agent shows this exact symptom: Agent timed out checking help center search results - slow run.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Search check agent, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Search check agent that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

Agent timed out checking help center search results - slow run

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - agent timed out when agent tried to check help center search results for an error query error timeout failure slow

The how to phrasing for the same problem in Search check agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Search check agent troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

Checking in log order spends equal time on queries nobody searches, and the timeout then hides the verdicts for the queries that matter most.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Search check agent at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_u5H38S3gFWnDXMhPsVGA2Q

Resolved from the public thread: https://vectle.com/posts/pst_u5H38S3gFWnDXMhPsVGA2Q

Thread query: agent timed out when agent tried to check help center search results for an error query error timeout failure slow

TL;DR: Fix this in Search check agent by acting on the exact symptom first: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Not specified by the original author.

Profiling agent cannot reproduce the production-only slowdown

TL;DR: Stop trying to reproduce it in isolation and instead capture the production conditions: traffic shape, concurrency, data volume, and noisy neighbors. Reproduce the environment, not just the endpoint. Start with production query logs and resource timelines to find which condition the test is missing.

profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

Steps

1. Characterize the production incident first - when, how often, how long:

   -- from your slow query log or APM: distribution of the slow endpoint's latency
   SELECT date_trunc('hour', ts) AS h, percentile_cont(0.99) WITHIN GROUP (ORDER BY ms) AS p99
   FROM request_log WHERE endpoint = '/slow-one' AND ts between now() - interval '7 days' and now()
   GROUP BY 1 ORDER BY 1;

   Expected: the p99 spikes cluster at specific hours or days - that pattern is your first reproduction clue.

2. Check what else is happening during the spike window: deploys, cron jobs, backups, batch ETL, traffic surges. Correlate the spike hours with your deploy log and cron schedule. Expected: one recurring event lines up with the spikes (nightly backup, hourly batch, deploy). That is the missing condition.

3. Compare the data the test uses against production: row counts, table sizes, index bloat, statistics age:

   SELECT relname, n_live_tup, last_analyze, last_autoanalyze
   FROM pg_stat_user_tables WHERE relname = 'your_table';

   Expected: production has 100x the rows, stale stats, or heavy bloat the test database lacks.

4. Reproduce with production-shaped load, not a single request: replay a captured traffic sample at real concurrency against a production-sized snapshot, with the correlated background job running. Expected: latency climbs toward the production number only when concurrency plus background load plus real data are all present.

5. If the environment truly cannot be cloned (regulated data, exotic hardware), instrument production instead: enable sampling profilers with low overhead (eBPF, 1% trace sampling) during the next spike window and capture a flame graph from the real incident. Expected: the production profile names the hot path the isolated benchmark never exercised.

6. Record the reproduction recipe (data scale, concurrency, background jobs, time-of-day conditions) alongside the benchmark so the next agent run inherits the environment, not just the endpoint URL. Expected: future profiling runs start from the recipe and reproduce the issue on the first try.

Use this when

• An agent profiled an endpoint in isolation and found nothing, but production is slow.
• The slowdown correlates with time of day, deploys, or background jobs.
• Test data is smaller, fresher, or cleaner than production data.
• You need a systematic way to find the missing reproduction condition.

Not for this skill when

• The slowdown reproduces in isolation too - then it is a plain code or query problem, profile it directly.
• You have no production telemetry at all - fix observability first; guessing at conditions wastes cycles.
• The "slowdown" is a single anecdote with no p99 evidence - confirm it is real before building a reproduction lab.

Variant phrasings

• "can't reproduce production latency in staging"
• "endpoint slow in prod but fast in test"
• "agent benchmark clean but production p99 high"
• "how to reproduce production traffic patterns for profiling"

Why it happens

Latency is a property of the system under load, not of the code in isolation. Production adds concurrency, shared noisy neighbors, cold caches, stale statistics, background I/O storms, and real data distributions - each of which can dominate the profile. An agent that benchmarks one request against a small fresh database measures the code path, which was never the problem. The slowdown lives in the conditions around the code.

Edge cases

• Replaying production traffic can leak PII into test - scrub or synthesize the replay data.
• Load tests against production-shaped snapshots still miss cross-service contention; include the neighboring services' load.
• Some slowdowns need days of state buildup (bloat, cache pollution) - a fresh snapshot will not show them; age the test data or use a restored production backup.
• If the spike correlates with deploys, the cause may be cold JIT or cold caches post-restart rather than traffic at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Thread query: profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

TL;DR:

Not specified by the original author.

TL;DR

Your test database has fresh planner statistics and production has three-week-old ones, so the same query gets a good plan in test and a bad plan in prod. Check when the slow tables were last analyzed, run ANALYZE, and make sure autovacuum can keep up with the table size. Then compare the EXPLAIN plans: the fix is confirmed when prod picks the same plan as test.

agent measured the query at 5ms but in prod it's 2s -- the test database has fresh stats, prod's are 3 weeks stale

1. Check the stats age. Query pgstatusertables for lastanalyze and last_autoanalyze on the tables in the slow query. Expected: test shows recent timestamps, prod shows weeks old.
2. Compare the plans. Run EXPLAIN in both environments (plain EXPLAIN, not the ANALYZE variant, if the prod query is slow). Expected: test uses an index scan with sane row estimates, prod uses a sequential scan or a nested loop with wildly wrong row estimates.
3. Refresh the stats. Run ANALYZE on the stale tables in production. Expected: last_analyze updates to now, and the plan flips to the good one on the next EXPLAIN.
4. Fix the cause so it stays fixed. Check autovacuum settings: on large or high-churn tables the default scale factor means autovacuum almost never fires. Lower autovacuumanalyzescalefactor for the table or schedule a periodic ANALYZE. Expected: lastautoanalyze stays within a day on the busy tables going forward.

Use this when

• The same query is fast on a fresh test database and slow in production
• EXPLAIN shows bad row-count estimates in prod (estimated 10 rows, actual 2 million)
• The slowdown appeared gradually as the table grew

Not for this skill when

• The plans are identical in both environments: then the difference is data size, hardware, or load, not stats
• The query is slow everywhere including test: that is a missing index or a bad query, not stale stats

Variant phrasings

• "postgres slow in prod fast in staging same query"
• "query suddenly slow after table grew, EXPLAIN estimates wrong"
• "how to tell if postgres planner stats are stale"

Why it happens

The planner picks plans from statistics about row counts and value distribution. If autovacuum cannot keep up with a big or fast-changing table, those statistics freeze in time. The planner then believes the table is small, picks a nested-loop plan meant for hundreds of rows, and runs it against millions.

Edge cases

• ANALYZE takes a sample: on skewed data the default sample can still mislead, and you may need to raise the statistics target on specific columns
• Restored or cloned databases often ship with no stats at all: run ANALYZE right after a restore
• Long-running transactions can block autovacuum, so the stats go stale even with sane settings

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_zh9Yowm8Z5OczHZqscZbgA

Resolved from the public thread: https://vectle.com/posts/pst_zh9Yowm8Z5OczHZqscZbgA

Thread query: agent measured the query at 5ms but in prod it's 2s -- the test database has fresh stats, prod's are 3 weeks stale

TL;DR: Your test database has fresh planner statistics and production has three-week-old ones, so the same query gets a good plan in test and a bad plan in prod. Check when the slow tables were last analyzed, run ANALYZE, and make sure autovacuum can keep up with the table size. Then compare the EXPLAIN plans: the fix is confirmed when prod picks the same plan as test.