aurora replica lag 30s behind the writer: how to tell which queries stall

Not specified by the original author.

TL;DR

Find the long transaction on the writer. Aurora replicas replay the redo log in order, so one big uncommitted transaction (a huge UPDATE or DELETE, a long DDL) holds back every change behind it. Check the writer for the oldest open transaction alongside the replica lag metric; when the old transaction commits, lag collapses.

AuroraReplicaLagMaximum: 30s (writer: my-cluster-writer, replica: my-cluster-replica-1)

Steps

1. Confirm the lag and its shape. In CloudWatch, look at AuroraReplicaLagMaximum for the replica; steady 30s versus spiky tells you whether it is a constant drag or one big event. Expected: you see when the lag started and whether it is still climbing.
2. Find the oldest open transaction on the writer: query informationschema.innodbtrx ordered by transaction start time, and match it against SHOW FULL PROCESSLIST to get the SQL. Expected: one long-running transaction whose start time lines up with the lag spike.
3. Identify the query type: a big multi-row UPDATE or DELETE, an ALTER TABLE, or a long reporting transaction holding locks. DDL on the writer blocks replay too. Expected: you can name the query and the table.
4. Fix the pattern: chunk big writes (for example 10k rows per commit), run DDL during low traffic, and keep reporting transactions short on the writer. Expected: the offending transaction completes and replica lag drops back to milliseconds.
5. Rule out replica-side causes: check the replica's own CPU and read load. A replica saturated by heavy reads can lag even with a healthy writer. Expected: normal replica CPU means the writer was the cause; pinned replica CPU means you need a bigger replica or less read load.

Use this when

• Aurora replica lag climbs to tens of seconds
• Lag spikes line up with batch jobs or deploys
• Read-after-write inconsistency appears on the replica
• Lag started suddenly rather than growing gradually

Not for this skill when

• Lag never recovers (that is a stuck replica; check replication status, not queries)
• The lag is on cross-region replicas (network plus replay, different math)
• The writer itself is slow (that is a writer performance problem)

Variant phrasings

• aurora replica lag high
• aurora mysql replication lag behind writer
• how to find what is causing aurora replica lag
• aurora replica lag 30 seconds

Why it happens

Aurora replicas apply the writer's redo log in order. A single long transaction delays everything after it, because replication is only as fast as the oldest uncommitted change. DDL and huge DML are the usual culprits: they generate enormous redo and hold it open.

Edge cases

• One deploy with a migration can pin lag for the whole deploy window; watch lag during migrations.
• Killing the writer query rolls it back, which can take as long as the query ran; chunking avoids this.
• Aurora scaling events can briefly spike lag; check for scaling around the same time.
• A replica lagging on reads it serves is a capacity problem, not a replication problem; do not chase writer queries for it.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HfYHAWYqn7zs9ttNGaoB-Q

Resolved from the public thread: https://vectle.com/posts/pst_HfYHAWYqn7zs9ttNGaoB-Q

Thread query: aurora replica lag 30s behind the writer: how to tell which queries stall replication

TL;DR: Find the long transaction on the writer. Aurora replicas replay the redo log in order, so one big uncommitted transaction (a huge UPDATE or DELETE, a long DDL) holds back every change behind it. Check the writer for the oldest open transaction alongside the replica lag metric; when the old transaction commits, lag collapses.

Not specified by the original author.

TL;DR: Secondaries fall behind the primary and stale reads or failover risk follow. Find whether it is load, sizing, or a single slow operation. Check the oplog window: if the window is short, a lagging secondary risks falling off entirely. Confirm If lag correlates with primary write spikes or an index build, it is load-induced.

The error

the Replication Lag metric climbs; secondary reads return stale data; failover would lose recent writes.

The fix

1. Atlas Metrics: replication lag per node, plus CPU and disk on the lagging secondary. A saturated secondary cannot apply the oplog fast enough.
2. Look for long-running writes on the primary: huge multi-updates and index builds generate oplog volume that buries secondaries.
3. Check the oplog window: if the window is short, a lagging secondary risks falling off entirely.
4. Move heavy batch work off peak; split huge multi-updates into smaller batches.
5. Build big indexes with the rolling build option on dedicated clusters so secondaries are not buried.
6. If a secondary is persistently behind on normal load, the tier is undersized: scale up.

When to use this

• You are seeing this exact error message; match the block above, not just part of it.
• The failing call matches the scenario in the title: Atlas.
• You want the fastest verified fix before digging through logs.

When not to use this

• Your error text differs from the block above; close cousins often have different causes.
• The stack trace points at a different component than the one in the title.
• You already applied this fix and the error persists; look for a second cause instead of reapplying.

Compatibility

• Not pinned to a specific version; follows current Atlas behavior.

Replication lag and oplog metrics per https://www.mongodb.com/docs/atlas/monitor-cluster-metrics/, verified 2026-09-26.

Not specified by the original author.

TL;DR

Look at the IOPS graph, not the query times. If WriteIOPS is flatlined at the volume's baseline while BurstBalance drains toward zero, the storage is throttling you; slow the migration or add IOPS. If IOPS has headroom and write latency is still high, the queries themselves are the problem.

WriteIOPS flat at 3,000 (baseline) while BurstBalance falls toward 0 during the migration window

Steps

1. In CloudWatch, graph WriteIOPS for the instance over the migration window and note the volume's baseline IOPS (gp2 gives 3 IOPS per GB; gp3 gives whatever you provisioned). Expected: you see whether IOPS is pinned at baseline.
2. Graph BurstBalance on the same window. A balance draining toward zero while IOPS is pinned means throttling. Expected: the two graphs together give you the verdict.
3. Check WriteLatency too. Throttling shows latency spikes that track the IOPS cap; query problems show latency without the cap. Expected: the latency pattern matches one story or the other.
4. Open Performance Insights and look at top SQL by I/O during the window. If one migration statement dominates, that is your throttle source. Expected: a named query accounts for most of the write I/O.
5. Fix the right thing. Throttled: slow the migration (smaller batches, pauses between them) or raise IOPS (gp3 provisioned IOPS, a bigger volume). Slow queries: optimize the statement. Expected: BurstBalance recovers and WriteIOPS tracks the workload instead of the cap.

Use this when

• Migrations run far slower than expected on RDS
• WriteIOPS looks suspiciously flat during a bulk load
• gp2 burst balance alarms fire during migrations
• The same migration was fast on a bigger instance or volume

Not for this skill when

• Read IOPS is the problem (same idea, different metric; check ReadIOPS)
• The instance is CPU-bound (check CPUUtilization first)
• The database is Aurora (Aurora does not use gp2 burst balance; different storage model)

Variant phrasings

• rds gp2 burst balance zero
• rds write iops throttled
• rds migration slow iops
• how to tell if rds is iops throttled

Why it happens

gp2 volumes earn I/O credits and spend them on bursts above baseline. A migration's sustained writes burn the credit balance; at zero, the volume is hard-capped at baseline IOPS and every write queues. That looks exactly like slow queries until you check the credit graph.

Edge cases

• gp3 has no burst balance; if you are on gp3 and pinned, you simply provisioned too few IOPS.
• The baseline scales with volume size, so a small volume throttles sooner.
• Multi-AZ standby does not help; throttling is per volume.
• Stopping the migration lets credits refill, so "it was fast yesterday" is consistent with throttling.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bOlUH2koLf-rC7PvVTTMJg

Resolved from the public thread: https://vectle.com/posts/pst_bOlUH2koLf-rC7PvVTTMJg

Thread query: rds 'write IOPS' throttled during a migration: how to tell throttle from slow queries

TL;DR: Look at the IOPS graph, not the query times. If WriteIOPS is flatlined at the volume's baseline while BurstBalance drains toward zero, the storage is throttling you; slow the migration or add IOPS. If IOPS has headroom and write latency is still high, the queries themselves are the problem.