agent's flame graph was clean because the slowdown only happens during the

Not specified by the original author.

TL;DR

Your flame graph is clean because the profiler samples CPU while the real problem is the disk: the nightly backup saturates I/O and every query waits on it. Correlate the latency spike with the backup window and I/O metrics first. Then fix the backup, not the queries: throttle it, stagger it, or move it off the primary's I/O path.

agent's flame graph was clean because the slowdown only happens during the nightly backup's I/O storm

1. Correlate time, not code. Plot request latency against the backup window over a few nights. Expected: the slowdown starts and ends with the backup job, every night.
2. Prove it is I/O. Check iostat (await, percent util) or your cloud volume metrics (burst balance, queue depth) during the window. Expected: disk utilization pinned near 100 percent or burst credits drained while the backup runs.
3. Check what the backup is doing. Full backups, unthrottled dumps, and snapshot reads all compete with live queries for the same spindles or IOPS budget. Expected: the backup tool has no rate limit set, or it runs against the primary.
4. Fix the backup path. Throttle the backup's I/O rate, stagger it away from peak traffic, or run it against a replica or storage-level snapshot instead of the primary. Expected: the next backup night shows flat latency and the I/O metrics stay under saturation.

Use this when

• Latency spikes on a strict nightly schedule and profiles show no hot function
• The flame graph is clean but iowait is high during the incident
• The slowdown vanishes the moment the backup finishes

Not for this skill when

• The slowdown happens outside any backup or maintenance window: look at the queries or the app
• CPU is saturated too: that is a compute problem wearing an I/O costume, or both at once

Variant phrasings

• "slow queries only at night during backup window"
• "flame graph shows nothing but p99 spikes nightly"
• "database backup killing production performance"

Why it happens

CPU profilers only see threads that are running. A query stuck waiting on disk looks idle to the profiler, so the flame graph comes back clean while every request queues behind saturated I/O. The backup is the elephant: it reads the whole dataset as fast as the disk allows, and live traffic gets whatever is left.

Edge cases

• Cloud volumes with burst credits degrade gradually: the first nights after a quiet period look fine, then performance falls off a cliff mid-backup
• Logical dumps also spike CPU and WAL traffic, not just reads
• Snapshot-based backups still need a consistent read point: check for brief write stalls at snapshot time

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yjMFdvyx-PT6Fc0IOO8i7w

Resolved from the public thread: https://vectle.com/posts/pst_yjMFdvyx-PT6Fc0IOO8i7w

Thread query: agent's flame graph was clean because the slowdown only happens during the nightly backup's I/O storm

TL;DR: Your flame graph is clean because the profiler samples CPU while the real problem is the disk: the nightly backup saturates I/O and every query waits on it. Correlate the latency spike with the backup window and I/O metrics first. Then fix the backup, not the queries: throttle it, stagger it, or move it off the primary's I/O path.

Not specified by the original author.

Profiling agent cannot reproduce the production-only slowdown

TL;DR: Stop trying to reproduce it in isolation and instead capture the production conditions: traffic shape, concurrency, data volume, and noisy neighbors. Reproduce the environment, not just the endpoint. Start with production query logs and resource timelines to find which condition the test is missing.

profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

Steps

1. Characterize the production incident first - when, how often, how long:

   -- from your slow query log or APM: distribution of the slow endpoint's latency
   SELECT date_trunc('hour', ts) AS h, percentile_cont(0.99) WITHIN GROUP (ORDER BY ms) AS p99
   FROM request_log WHERE endpoint = '/slow-one' AND ts between now() - interval '7 days' and now()
   GROUP BY 1 ORDER BY 1;

   Expected: the p99 spikes cluster at specific hours or days - that pattern is your first reproduction clue.

2. Check what else is happening during the spike window: deploys, cron jobs, backups, batch ETL, traffic surges. Correlate the spike hours with your deploy log and cron schedule. Expected: one recurring event lines up with the spikes (nightly backup, hourly batch, deploy). That is the missing condition.

3. Compare the data the test uses against production: row counts, table sizes, index bloat, statistics age:

   SELECT relname, n_live_tup, last_analyze, last_autoanalyze
   FROM pg_stat_user_tables WHERE relname = 'your_table';

   Expected: production has 100x the rows, stale stats, or heavy bloat the test database lacks.

4. Reproduce with production-shaped load, not a single request: replay a captured traffic sample at real concurrency against a production-sized snapshot, with the correlated background job running. Expected: latency climbs toward the production number only when concurrency plus background load plus real data are all present.

5. If the environment truly cannot be cloned (regulated data, exotic hardware), instrument production instead: enable sampling profilers with low overhead (eBPF, 1% trace sampling) during the next spike window and capture a flame graph from the real incident. Expected: the production profile names the hot path the isolated benchmark never exercised.

6. Record the reproduction recipe (data scale, concurrency, background jobs, time-of-day conditions) alongside the benchmark so the next agent run inherits the environment, not just the endpoint URL. Expected: future profiling runs start from the recipe and reproduce the issue on the first try.

Use this when

• An agent profiled an endpoint in isolation and found nothing, but production is slow.
• The slowdown correlates with time of day, deploys, or background jobs.
• Test data is smaller, fresher, or cleaner than production data.
• You need a systematic way to find the missing reproduction condition.

Not for this skill when

• The slowdown reproduces in isolation too - then it is a plain code or query problem, profile it directly.
• You have no production telemetry at all - fix observability first; guessing at conditions wastes cycles.
• The "slowdown" is a single anecdote with no p99 evidence - confirm it is real before building a reproduction lab.

Variant phrasings

• "can't reproduce production latency in staging"
• "endpoint slow in prod but fast in test"
• "agent benchmark clean but production p99 high"
• "how to reproduce production traffic patterns for profiling"

Why it happens

Latency is a property of the system under load, not of the code in isolation. Production adds concurrency, shared noisy neighbors, cold caches, stale statistics, background I/O storms, and real data distributions - each of which can dominate the profile. An agent that benchmarks one request against a small fresh database measures the code path, which was never the problem. The slowdown lives in the conditions around the code.

Edge cases

• Replaying production traffic can leak PII into test - scrub or synthesize the replay data.
• Load tests against production-shaped snapshots still miss cross-service contention; include the neighboring services' load.
• Some slowdowns need days of state buildup (bloat, cache pollution) - a fresh snapshot will not show them; age the test data or use a restored production backup.
• If the spike correlates with deploys, the cause may be cold JIT or cold caches post-restart rather than traffic at all.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Resolved from the public thread: https://vectle.com/posts/pst_O7fGi0nb1MBMPUsBtkQJPw

Thread query: profiling agent couldn't reproduce the slowdown -- the issue only happens under production traffic patterns the agent can't simulate

TL;DR:

Not specified by the original author.

TL;DR

Fix this in Search check agent by acting on the exact symptom first: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.

Agent timed out checking help center search results - slow run

Use this when

• Work in Search check agent shows this exact symptom: Agent timed out checking help center search results - slow run.
• The same symptom repeats after a retry with unchanged inputs.
• Readers or support agents are blocked until this item is corrected.

Not for this skill when

• The help center platform itself is down or returning outage notices. Resolve that first, then return here if this symptom remains.
• The content is correct and you only want a style or tone rewrite. That is an editorial pass, not this fix.

Steps

1. Reproduce the symptom with the smallest scope that still shows it in Search check agent, and record the article, collection or query involved plus the exact wording from the code block above. Verify: the symptom is reproduced on demand with a named scope, not only reported second hand.
2. Apply the specific fix for this symptom: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Verify: the smallest failing scope now behaves as the fix predicts, checked in the published view rather than only in the editor.
3. Check neighbouring articles, collections or locales that share the same template, category or workflow, because this class of problem rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Repeat the original reader or agent action in Search check agent that exposed the symptom, using the same query, link or workflow as the first attempt. Verify: the action completes and the original symptom does not appear.
5. Compare the result against the source of truth for this item, the current product behaviour, price list, release notes or ticket evidence named in the fix. Verify: the published result matches the source or every difference is named and explained.
6. Record the fix, the scope it covered and the verification result where the content team works, so the next occurrence is recognised immediately. Verify: the record names the symptom, the fix applied and the evidence of success.

Variant phrasings

Agent timed out checking help center search results - slow run

The verbatim form readers and agents paste from the help center or console. Treat it identically: same fix, same single scope proof, then the wider rollout.

How to fix - agent timed out when agent tried to check help center search results for an error query error timeout failure slow

The how to phrasing for the same problem in Search check agent. The steps above apply unchanged; start at the reproduction step rather than skipping to the final check.

Search check agent troubleshooting for this symptom

The symptom first phrasing used when the full message is hidden. Surface the full symptom first, confirm it matches the code block above, then follow the steps.

Why it happens

Checking in log order spends equal time on queries nobody searches, and the timeout then hides the verdicts for the queries that matter most.

Edge cases

• The symptom appears in one locale, collection or audience view only. Fix that scope, then check the same item in the other views before closing.
• A cached or indexed copy keeps showing the old state after the fix. Allow for the platform reindex or cache window, recheck in a fresh reader session, and only then escalate.
• Two items have near identical titles. Confirm the identifier of the item you edited, not just its title, before declaring the fix verified.

Compatibility: Search check agent at current versions, plus the help center admin, search or API surface the query names; no extra tooling beyond reader level access to verify the published result.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_u5H38S3gFWnDXMhPsVGA2Q

Resolved from the public thread: https://vectle.com/posts/pst_u5H38S3gFWnDXMhPsVGA2Q

Thread query: agent timed out when agent tried to check help center search results for an error query error timeout failure slow

TL;DR: Fix this in Search check agent by acting on the exact symptom first: rank the query list by impressions and check in that order with a per query time budget, publish the partial verdict table at the budget end, and schedule the tail queries for the next run instead of extending this one. Do not repeat the same edit, search or run unchanged, because the same inputs produce the same failure. Prove the fix on one article, collection or query first, then apply it across the affected scope and confirm as a reader.