offsite replication lag keeps growing backup never catches up

TL;DR

Measure first: compare the daily change rate being replicated with the effective throughput the link and target actually sustain during the replication window. If change exceeds throughput, no schedule tweak will catch up - you must seed, widen the window, throttle competing traffic, or reduce change. Fix the measured bottleneck, then watch lag fall for two full cycles before declaring recovery.

offsite replication lag keeps growing backup never catches up

Use this when

• Offsite or replica copy is always hours or days behind and the gap grows each day.
• Replication jobs run to completion but the newest replicated point gets older, not newer.
• A bandwidth, target ingest, or seeding limit is suspected but not proven.

Not for this skill when

• Replication fails with an authentication, snapshot, or target full error. Fix that named error first.
• Lag is a one off after an outage and is already shrinking each cycle. Monitor instead of redesigning.

Steps

1. Calculate lag in data and in time for seven days: change generated per day, bytes replicated per day, and the age of the newest completed replica point each morning. Verify: the daily shortfall, replicated minus changed, is a specific number, not a feeling
2. Measure effective throughput during the actual window, at the source egress, across the link, and into target ingest, and note which is lowest. Include target write limits, cloud ingest throttling, and any bandwidth schedule on the backup product. Verify: one bottleneck stage is identified with its measured rate
3. Check for hidden change inflation: a workload that rewrites whole files, database maintenance, antivirus touches, or a job replicating both local backup and replica traffic over the same link. Verify: the top changing systems are listed by bytes per day
4. Remove fixable waste: exclude temporary and cache data, stop replicating superseded points that retention no longer needs, deduplicate or compress before transfer where the product supports it, and align windows so backup and replication do not compete. Verify: change bytes per day drop, or window throughput rises, in the next cycle measurement
5. If the shortfall is structural, reseed: ship or provision a seed copy by local copy, removable media, or a one time high bandwidth transfer, then restart replication from the seed so only daily change crosses the link. Verify: after seeding, the newest replica point is current within one cycle
6. Set a lag based alert on replica point age and shortfall trend, and review change rate weekly so growth is caught before the gap becomes unrecoverable. Verify: a test alert fires when replica age crosses your recovery point objective

Why it happens

Replication lag is arithmetic before it is a product bug. If a system changes 800 GB a day and the window, link, throttling policy, and target ingest together sustain 500 GB a day, the backlog grows by 300 GB daily and the replica can never catch up, even with every job reporting success. Throughput is often far below headline link speed because of latency on many small files, single threaded transfer streams, target side dedupe or checksum work, cloud rate limits, and bandwidth schedules set years ago for a smaller data set. Change inflation makes it worse: a single database or virtual disk that rewrites its full size nightly counts as full change. Seeding breaks the arithmetic by moving the bulk once by another path, leaving the link with only the daily delta it can actually sustain.

Variants and edge cases

lag grows only on weekdays

Weekday change from users and weekday database maintenance exceeds capacity while weekends catch up partially. Either extend the weekday window, replicate critical systems continuously and bulk systems at weekends, or split the fleet across two targets or links.

bandwidth looks free but transfer is slow

High latency links with many small objects or a single transfer stream underuse bandwidth. Enable parallel streams, larger block or object sizes, and local staging where the product supports them, and measure again - headline utilisation can stay low while throughput doubles.

target ingest is the bottleneck

Cloud and appliance targets throttle ingest, run their own dedupe and integrity work, and slow down when near full. Check target health, free space, and vendor rate limits before upgrading the link, because a faster link into a saturated target changes nothing. Edge cases: retention on the target can pin old points and fill the target, slowing ingest for new data, so check target capacity alongside transfer rates. Clock and reporting traps matter too: some consoles show job start lag rather than newest completed point age, hiding real risk. Always report lag as the age of the newest restore ready replica, because that is the number that defines data loss in a site failure, and trend it daily until it stays inside the recovery point objective for a full week.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__Ra8QQbd5Cmcp-SfOT-krw