agent shrank the database volume below the provisioned IOPS the workload needed, latency spiked
Separates volume size from provisioned IOPS when downsizing database storage so throughput never drops with the disk. Use when a rightsizing run shrank an EBS volume and database latency spiked. Not for gp2 volumes where IOPS are tied to size.
TL;DR
On gp3 and io2, size and IOPS are independent knobs. When downsizing a database volume, keep the provisioned IOPS at or above the workload's measured peak, and only shrink the gigabytes. Check the workload's actual IOPS first; the volume size was never the number that mattered.
agent shrank the database volume below the provisioned IOPS the workload needed, latency spikedUse this when
- Database latency spiked right after a volume downsizing.
- The volume was shrunk on size while IOPS were left at the old ratio or lowered.
- The rightsizing script treated storage as one number.
Not for this skill when
- The volume is gp2. There IOPS scale with size, so shrinking the disk genuinely lowers IOPS and the fix is different (migrate to gp3 first).
- Latency spiked for another reason (query plan, connection pool). Confirm the timing lines up with the resize.
Steps
- Measure the workload's real IOPS: CloudWatch
VolumeReadOps+VolumeWriteOpsover the last 30 days, and note the peak. Verify: you have a peak number, not an average. - Read the current volume config:
aws ec2 describe-volumes. Verify: size and IOPS are what you expect after the shrink. - Raise IOPS back to at least the measured peak (add headroom, 20-30%) without changing size:
aws ec2 modify-volume --volume-id VOL_ID --iops NEW_VALUE. - Watch latency for a full workload cycle. Verify: p99 latency returns to the pre-shrink baseline.
- Fix the rightsizing script: size and IOPS must be evaluated as separate knobs, with IOPS floored at measured peak plus headroom.
- Consider throughput too: gp3 throughput (MiB/s) is a third knob and can bottleneck the same way.
Variant phrasings
ebs downsized latency spike iops
Symptom-level search.
gp3 modify volume iops without resizing
The fix command framing.
database slow after shrinking ebs volume
How the on-call describes it.
Compatibility: gp3 and io1/io2 support independent IOPS; gp2 does not. Database must be stopped or support online resize depending on engine and filesystem.
Why it happens
Rightsizing scripts see "200 GB used of 500 GB" and shrink to 250 GB, dragging the IOPS assumption down with it or leaving IOPS at a value that fit the old size. Storage size was never the performance number; IOPS was. The database notices within one peak window.
Edge cases / pitfalls
- Raising IOPS costs money per provisioned IOPS. Price the IOPS floor before committing; sometimes the smaller disk plus higher IOPS costs more than the original.
- Some filesystems need expansion steps after a shrink to stay healthy; verify the OS sees the volume correctly post-change.
- io2 to gp3 migrations are usually cheaper for the same IOPS; evaluate the volume type while you are touching it.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst8Romb10dc8rwcuQylCIAg