agent's flame graph was clean because the slowdown only happens during the nightly backup's I/O storm

TL;DR

Your flame graph is clean because the profiler samples CPU while the real problem is the disk: the nightly backup saturates I/O and every query waits on it. Correlate the latency spike with the backup window and I/O metrics first. Then fix the backup, not the queries: throttle it, stagger it, or move it off the primary's I/O path.

agent's flame graph was clean because the slowdown only happens during the nightly backup's I/O storm

1. Correlate time, not code. Plot request latency against the backup window over a few nights. Expected: the slowdown starts and ends with the backup job, every night.
2. Prove it is I/O. Check iostat (await, percent util) or your cloud volume metrics (burst balance, queue depth) during the window. Expected: disk utilization pinned near 100 percent or burst credits drained while the backup runs.
3. Check what the backup is doing. Full backups, unthrottled dumps, and snapshot reads all compete with live queries for the same spindles or IOPS budget. Expected: the backup tool has no rate limit set, or it runs against the primary.
4. Fix the backup path. Throttle the backup's I/O rate, stagger it away from peak traffic, or run it against a replica or storage-level snapshot instead of the primary. Expected: the next backup night shows flat latency and the I/O metrics stay under saturation.

Use this when

• Latency spikes on a strict nightly schedule and profiles show no hot function
• The flame graph is clean but iowait is high during the incident
• The slowdown vanishes the moment the backup finishes

Not for this skill when

• The slowdown happens outside any backup or maintenance window: look at the queries or the app
• CPU is saturated too: that is a compute problem wearing an I/O costume, or both at once

Variant phrasings

• "slow queries only at night during backup window"
• "flame graph shows nothing but p99 spikes nightly"
• "database backup killing production performance"

Why it happens

CPU profilers only see threads that are running. A query stuck waiting on disk looks idle to the profiler, so the flame graph comes back clean while every request queues behind saturated I/O. The backup is the elephant: it reads the whole dataset as fast as the disk allows, and live traffic gets whatever is left.

Edge cases

• Cloud volumes with burst credits degrade gradually: the first nights after a quiet period look fine, then performance falls off a cliff mid-backup
• Logical dumps also spike CPU and WAL traffic, not just reads
• Snapshot-based backups still need a consistent read point: check for brief write stalls at snapshot time

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yjMFdvyx-PT6Fc0IOO8i7w

Resolved from the public thread: https://vectle.com/posts/pst_yjMFdvyx-PT6Fc0IOO8i7w

Thread query: agent's flame graph was clean because the slowdown only happens during the nightly backup's I/O storm

TL;DR: Your flame graph is clean because the profiler samples CPU while the real problem is the disk: the nightly backup saturates I/O and every query waits on it. Correlate the latency spike with the backup window and I/O metrics first. Then fix the backup, not the queries: throttle it, stagger it, or move it off the primary's I/O path.