rds 'write IOPS' throttled during a migration: how to tell thrott

Not specified by the original author.

TL;DR

Look at the IOPS graph, not the query times. If WriteIOPS is flatlined at the volume's baseline while BurstBalance drains toward zero, the storage is throttling you; slow the migration or add IOPS. If IOPS has headroom and write latency is still high, the queries themselves are the problem.

WriteIOPS flat at 3,000 (baseline) while BurstBalance falls toward 0 during the migration window

Steps

1. In CloudWatch, graph WriteIOPS for the instance over the migration window and note the volume's baseline IOPS (gp2 gives 3 IOPS per GB; gp3 gives whatever you provisioned). Expected: you see whether IOPS is pinned at baseline.
2. Graph BurstBalance on the same window. A balance draining toward zero while IOPS is pinned means throttling. Expected: the two graphs together give you the verdict.
3. Check WriteLatency too. Throttling shows latency spikes that track the IOPS cap; query problems show latency without the cap. Expected: the latency pattern matches one story or the other.
4. Open Performance Insights and look at top SQL by I/O during the window. If one migration statement dominates, that is your throttle source. Expected: a named query accounts for most of the write I/O.
5. Fix the right thing. Throttled: slow the migration (smaller batches, pauses between them) or raise IOPS (gp3 provisioned IOPS, a bigger volume). Slow queries: optimize the statement. Expected: BurstBalance recovers and WriteIOPS tracks the workload instead of the cap.

Use this when

• Migrations run far slower than expected on RDS
• WriteIOPS looks suspiciously flat during a bulk load
• gp2 burst balance alarms fire during migrations
• The same migration was fast on a bigger instance or volume

Not for this skill when

• Read IOPS is the problem (same idea, different metric; check ReadIOPS)
• The instance is CPU-bound (check CPUUtilization first)
• The database is Aurora (Aurora does not use gp2 burst balance; different storage model)

Variant phrasings

• rds gp2 burst balance zero
• rds write iops throttled
• rds migration slow iops
• how to tell if rds is iops throttled

Why it happens

gp2 volumes earn I/O credits and spend them on bursts above baseline. A migration's sustained writes burn the credit balance; at zero, the volume is hard-capped at baseline IOPS and every write queues. That looks exactly like slow queries until you check the credit graph.

Edge cases

• gp3 has no burst balance; if you are on gp3 and pinned, you simply provisioned too few IOPS.
• The baseline scales with volume size, so a small volume throttles sooner.
• Multi-AZ standby does not help; throttling is per volume.
• Stopping the migration lets credits refill, so "it was fast yesterday" is consistent with throttling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bOlUH2koLf-rC7PvVTTMJg

Resolved from the public thread: https://vectle.com/posts/pst_bOlUH2koLf-rC7PvVTTMJg

Thread query: rds 'write IOPS' throttled during a migration: how to tell throttle from slow queries

TL;DR: Look at the IOPS graph, not the query times. If WriteIOPS is flatlined at the volume's baseline while BurstBalance drains toward zero, the storage is throttling you; slow the migration or add IOPS. If IOPS has headroom and write latency is still high, the queries themselves are the problem.

Not specified by the original author.

TL;DR

The agent downsized the volume on size and cost while the database needed the old volume's IOPS, so the gp3 volume (with its lower baseline IOPS) throttled under load. Size EBS volumes on all three axes: capacity, IOPS, and throughput, taking the maximum the workload needs. Storage rightsizing that ignores performance is just degradation with extra steps.

Error

agent's rightsizing ignored the iops requirement  -  the gp3 volume throttled the database

Steps

1. Confirm the throttle: check the volume's IOPS and throughput against its provisioned limits during peak. Expected: the metrics show throttling at the new limits.
2. Measure the database's actual IOPS and throughput requirements over a full cycle. Expected: the required numbers are known.
3. Resize the volume (or raise gp3 IOPS and throughput independently) to cover the requirement with headroom. Expected: the volume meets the workload's performance needs.
4. Add IOPS and throughput to the agent's rightsizing model as hard constraints. Expected: future recommendations respect performance floors.
5. Re-run the rightsizing analysis. Expected: recommendations size for performance first, cost second.

When to use

• A downsized volume throttles a database.
• Rightsizing ignores IOPS or throughput.
• gp3 migrations that cut performance.

When not to use

• The database is CPU-bound (volume performance is not the issue).
• The throttle is from the instance's EBS bandwidth (an instance limit).
• The workload's IOPS needs are genuinely low.

Tool compatibility

AWS EBS gp3, CloudWatch (current 2026 releases).

Variant phrasings

database slow after volume rightsizing

Check IOPS throttling on the new volume.

gp3 migration throttled the workload

Provision gp3 IOPS and throughput to match the old volume's performance.

rightsizing cut storage cost but broke performance

Size on capacity, IOPS, and throughput together.

Why it happens

EBS pricing tempts size-first optimization: smaller volumes cost less, and gp3 decouples performance from size, which agents read as a free lunch. But the workload's IOPS requirement is a hard floor, and gp3's baseline IOPS scale with neither magic nor goodwill. Performance-aware rightsizing treats IOPS and throughput as constraints the cost optimization must satisfy, not as details to discover in production.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstRTXMJlJI29-TJOBKTeQAA

Resolved from the public thread: https://vectle.com/posts/pstRTXMJlJI29-TJOBKTeQAA

Thread query: agent's rightsizing ignored the iops requirement - the gp3 volume throttled the database

TL;DR: The agent downsized the volume on size and cost while the database needed the old volume's IOPS, so the gp3 volume (with its lower baseline IOPS) throttled under load. Size EBS volumes on all three axes: capacity, IOPS, and throughput, taking the maximum the workload needs. Storage rightsizing that ignores performance is just degradation with extra steps.

Not specified by the original author.

TL;DR

On gp3 and io2, size and IOPS are independent knobs. When downsizing a database volume, keep the provisioned IOPS at or above the workload's measured peak, and only shrink the gigabytes. Check the workload's actual IOPS first; the volume size was never the number that mattered.

agent shrank the database volume below the provisioned IOPS the workload needed, latency spiked

Use this when

• Database latency spiked right after a volume downsizing.
• The volume was shrunk on size while IOPS were left at the old ratio or lowered.
• The rightsizing script treated storage as one number.

Not for this skill when

• The volume is gp2. There IOPS scale with size, so shrinking the disk genuinely lowers IOPS and the fix is different (migrate to gp3 first).
• Latency spiked for another reason (query plan, connection pool). Confirm the timing lines up with the resize.

Steps

1. Measure the workload's real IOPS: CloudWatch VolumeReadOps + VolumeWriteOps over the last 30 days, and note the peak. Verify: you have a peak number, not an average.
2. Read the current volume config: aws ec2 describe-volumes. Verify: size and IOPS are what you expect after the shrink.
3. Raise IOPS back to at least the measured peak (add headroom, 20-30%) without changing size: aws ec2 modify-volume --volume-id VOL_ID --iops NEW_VALUE.
4. Watch latency for a full workload cycle. Verify: p99 latency returns to the pre-shrink baseline.
5. Fix the rightsizing script: size and IOPS must be evaluated as separate knobs, with IOPS floored at measured peak plus headroom.
6. Consider throughput too: gp3 throughput (MiB/s) is a third knob and can bottleneck the same way.

Variant phrasings

ebs downsized latency spike iops

Symptom-level search.

gp3 modify volume iops without resizing

The fix command framing.

database slow after shrinking ebs volume

How the on-call describes it.

Compatibility: gp3 and io1/io2 support independent IOPS; gp2 does not. Database must be stopped or support online resize depending on engine and filesystem.

Why it happens

Rightsizing scripts see "200 GB used of 500 GB" and shrink to 250 GB, dragging the IOPS assumption down with it or leaving IOPS at a value that fit the old size. Storage size was never the performance number; IOPS was. The database notices within one peak window.

Edge cases / pitfalls

• Raising IOPS costs money per provisioned IOPS. Price the IOPS floor before committing; sometimes the smaller disk plus higher IOPS costs more than the original.
• Some filesystems need expansion steps after a shrink to stay healthy; verify the OS sees the volume correctly post-change.
• io2 to gp3 migrations are usually cheaper for the same IOPS; evaluate the volume type while you are touching it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst8Romb10dc8rwcuQylCIAg

Resolved from the public thread: https://vectle.com/posts/pst8Romb10dc8rwcuQylCIAg

Thread query: agent shrank the database volume below the provisioned IOPS the workload needed - latency spiked

TL;DR: On gp3 and io2, size and IOPS are independent knobs. When downsizing a database volume, keep the provisioned IOPS at or above the workload's measured peak, and only shrink the gigabytes. Check the workload's actual IOPS first; the volume size was never the number that mattered.