incremental backup chain broken after failed merge

Not specified by the original author.

TL;DR

Stop the job from retrying the merge, identify the last restore point that passes a health check, and protect it before any cleanup. A merge interrupted by space loss, a repository outage or a killed process leaves the full file and increments inconsistent, so start a new active full or a fresh chain from the good point rather than deleting increments at random. Then fix the interruption cause so the next merge completes.

incremental backup chain broken after failed merge

Use this when

• An incremental job fails at merge, transform or synthetic full stage and later points will not restore.
• The product reports a broken or incomplete chain after a merge was interrupted.
• A repository ran out of space or a backup server restarted during a merge.

Not for this skill when

• One increment file was manually deleted and the product has a documented forget or repair flow for exactly that case. Follow that flow.
• The source VM changed hosts and the job mapping broke. That is a job mapping problem before it is a chain problem.
• All restore points including the base full fail health checks. That suggests repository wide damage needing recovery work.

Steps

1. Disable the schedule for the affected job and record the chain layout from the repository and the product catalog: the base full file, each increment in order, their sizes and timestamps, and which point the failed merge was creating. Compare the catalog view with the files actually on disk and note any size or timestamp that stops mid pattern. Verify: you have a chain map showing the last point the catalog trusts and the exact point where files and catalog diverge
2. Run the product health check or verification against restore points working backwards from newest to oldest until one passes cleanly. Treat the newest passing point as your protected recovery position and copy or export it if policy and space allow, before any chain surgery. Verify: one named restore point is verified good, is marked protected, and no cleanup step can remove it
3. Reconstruct the interruption from logs and infrastructure events at the merge time: repository free space history, backup server restarts, antivirus scans, storage latency or a process kill. Merges rewrite large amounts of data, so a transient space or outage event at that moment is the usual trigger. Verify: the interruption cause is identified with a timestamp that matches the failed merge in the job log
4. Fix that cause first, for example freeing repository space well above the merge working requirement, stabilising the repository connection, or moving the merge window away from other heavy jobs. Check the repository filesystem for errors if the interruption involved storage faults. Verify: repository free space and health checks pass with headroom greater than the largest full file the merge must rewrite
5. Rebuild according to the product rules: run an active full backup to start a clean chain, or use the documented chain repair or compact operation if your product provides one and support guidance matches your state. Never rename, renumber or delete increment files by hand to make the catalog look tidy. Verify: a new full restore point completes, passes a health check, and the job catalog shows a clean unbroken chain from that point
6. Test restore one file and, for a VM, a boot from the new full and from the protected older point if it was retained. Then re enable the schedule with merge monitoring for the first cycle, alerting on merge stage failures specifically rather than only on final job status. Verify: restore tests pass from the retained points, the next incremental and its merge complete, and the chain health check is clean

Why it happens

Forever incremental designs keep one full backup and transform increments into it during merge, updating catalog records as blocks move. If that process is interrupted part way, some blocks and records reflect the new state and some the old, so the chain no longer describes a single consistent point in time. Later increments built on that state inherit the problem, which is why the job may keep running while restores fail. Deleting the odd looking files makes it worse because the catalog still references them. The safe path is always to anchor on a verified point, remove the interruption cause, and rebuild the chain with product operations.

Variants and edge cases

Retention driven merges fail quietly for weeks on some setups because the job status shows success with warnings. Add a specific alert for merge and transform warnings and audit chain health monthly, not only after a restore fails. Also plan repository space for merge working room, not just for the final chain size; merges need both states present at once.

veeam forever forward incremental merge failed chain broken

The Veeam phrasing. Look for the transform or merge lines in the job session and check repository space at that timestamp before starting an active full.

incremental chain broken after repository out of space during merge

Use when space is the proven interruption. Freeing space alone does not repair the half merged chain; rebuild from the verified point.

backup chain health check failed after failed synthetic full

The synthetic variant. The same logic applies when a synthetic full assembly, not a retention merge, was interrupted by a proxy or repository fault.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCOwb0A6s3cLrINEXVM4gA

Resolved from the public thread: https://vectle.com/posts/pstCOwb0A6s3cLrINEXVM4gA

Thread query: incremental backup chain broken after failed merge

TL;DR: Stop the job from retrying the merge, identify the last restore point that passes a health check, and protect it before any cleanup. A merge interrupted by space loss, a repository outage or a killed process leaves the full file and increments inconsistent, so start a new active full or a fresh chain from the good point rather than deleting increments at random. Then fix the interruption cause so the next merge completes.

Not specified by the original author.

TL;DR

Compare the incremental failure log line by line with a successful full; the difference is the diagnosis, because incrementals depend on changed block tracking, prior chain state and snapshot or merge steps a full does not need. Reset or repair the tracking, clear the chain or snapshot limit problem the log names, then run an incremental in daylight. A full backup working is proof the source, network and target basics are fine, so do not rebuild them.

full backup succeeds but incremental fails every night

Use this when

• Full backups complete but incremental backups fail on a predictable nightly pattern.
• The incremental fails at tracking, snapshot, chain or merge stage rather than at initial connection.
• The pattern started after an upgrade, a VM migration or a storage change.

Not for this skill when

• Full and incremental both fail with the same target or credential error. Fix the shared cause first.
• The incremental is skipped by schedule design on some nights. Check the job calendar before treating it as a failure.
• A database native incremental or log backup is failing. That workflow has its own chain rules.

Steps

1. Collect one failed incremental log and one successful full log for the same VM or data set and compare stage by stage. Mark the first stage present in the incremental that the full does not run, such as changed block tracking read, previous chain open, snapshot retention check or merge. Verify: the comparison names one incremental only stage as the failure point instead of a generic job failed label
2. If tracking is named, check the changed block tracking state for the VM or volume: whether it is enabled, whether the hypervisor reset it after a migration or power event, and whether the driver or CBT files show errors. Reset tracking through the product documented method so the next run can establish a clean baseline, accepting it may take a larger first pass. Verify: tracking shows enabled and healthy after the reset, and the product log for the test run no longer reports tracking errors
3. If chain or merge is named, run the chain health check, confirm repository space covers merge working room, and look for the previous incremental that first warned. A chain damaged days ago makes every later incremental fail while a fresh full, which ignores the old chain, sails through. Verify: the health check either passes or identifies the damaged point, and repository space exceeds the merge requirement with margin
4. If snapshot limits are named, check shadow copy or snapshot count and storage caps on the source volume, old orphaned snapshots from failed runs, and retention settings that only incremental runs hit because fulls create and remove theirs differently. Clear orphans through the hypervisor or operating system tools, not by deleting snapshot files directly. Verify: snapshot counts are below the platform limit, no orphaned backup snapshots remain, and a manual snapshot and delete cycle succeeds
5. Run an incremental manually during the day with the same job settings and watch the previously failing stage. If the daylight run passes but nights fail, investigate what else runs at night, such as antivirus scans, patching reboots, or another job locking the VM or saturating the repository at the same hour. Verify: the manual incremental completes, or the night only condition is identified with a conflicting task or event at the same time
6. Put the job back on schedule and monitor three consecutive incrementals, not one, because tracking resets and chain fixes can pass once and fail at the next merge. Record which incremental only stage caused the original pattern in the runbook. Verify: three scheduled incrementals complete in a row and the runbook documents the mode specific cause and its check

Why it happens

Full and incremental backups share the easy parts: connecting, reading data at least once, and writing to the target. Incrementals add dependencies a full can bypass, namely a valid previous chain, changed block tracking that survived since the last run, snapshot bookkeeping within platform limits, and often a merge or synthetic step at the end. Any one of those can be broken while fulls keep succeeding, which is actually useful evidence that the basics are sound. Nightly predictability usually means a state problem that every incremental inherits, or a night only neighbour job, rather than random hardware faults.

Variants and edge cases

Weekday patterns matter. A Monday incremental after a weekend patch cycle carries far more change and can exceed snapshot storage or the window while weekday runs pass, or the reverse if weekend fulls reset the chain. Plot failure against change rate and patch calendar before changing product settings. Also check retention: an incremental that must merge the oldest point needs that point present, so an aggressive cleanup job can break incrementals alone.

veeam incremental fails but full works cbt reset

The tracking variant. After host migrations or power events, CBT data can be invalid and every incremental fails until tracking is reset.

incremental backup fails at merge every night full succeeds

Use when the read phase completes and only the closing merge fails. Focus on chain health and repository working space.

windows incremental backup fails shadow copy limit

The snapshot limit variant. The source volume snapshot storage or count cap is consumed by earlier failed runs, so each new incremental fails at creation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uqZD9E3SatjVH-hPLd668g

Resolved from the public thread: https://vectle.com/posts/pst_uqZD9E3SatjVH-hPLd668g

Thread query: full backup succeeds but incremental fails every night

TL;DR: Compare the incremental failure log line by line with a successful full; the difference is the diagnosis, because incrementals depend on changed block tracking, prior chain state and snapshot or merge steps a full does not need. Reset or repair the tracking, clear the chain or snapshot limit problem the log names, then run an incremental in daylight. A full backup working is proof the source, network and target basics are fine, so do not rebuild them.

Not specified by the original author.

TL;DR

Do not trust the passing verification until you know exactly what it tested: most verification only checks block checksums or mounts a file system, it does not boot the system, start the application, or replay the full chain a real restore needs. Reproduce the real restore in isolation, compare it step for step with the verification path, and widen verification to cover the failing layer.

backup verification passes but actual restore fails

Use this when

• Verification or health checks pass, but a real restore fails, boots badly, or returns corrupt data.
• Verification tests the newest point only while the failed restore used an older or longer chain.
• A file level verification passes for a system that must boot or run a database.

Not for this skill when

• Verification itself reports checksum or repository errors. Fix the repository fault first.
• The restore failed for a one off target hardware or driver reason. Use the dissimilar hardware skill.

Steps

1. Write down exactly what the passing verification does in your product: checksum only, file system mount, boot test, or application test, and which restore point and chain length it used. Verify: the verification scope is stated in one sentence and matches its log, not its job name
2. Repeat the failed real restore into an isolated target, capturing where it stops: chain assembly, data read, boot, service start, or application open. Verify: the failure layer is named with the exact error and timestamp from the restore log
3. Restore the exact point verification tested and the exact point that failed in production, side by side, and compare chain length, included disks, and application consistency settings. Verify: any difference in point, chain, or options between the two runs is listed
4. Run a full chain test: apply the full plus every incremental up to the target point instead of letting verification synthesize or shortcut the chain. Verify: every chain link is read during the test and a damaged middle link, if present, is identified by date
5. Add the missing layer to verification: a boot test for system images, a database open and query test for database backups, and a sample file hash comparison for file backups. Verify: the widened test fails when you deliberately point it at the known bad restore point
6. Schedule the widened verification across the retention window, not only on the newest point, and keep its results per point so a failing point is quarantined. Verify: older points appear in the report with individual pass or fail results

Why it happens

Verification and restore exercise different code paths. A checksum verification proves stored blocks match the hashes recorded when they were written; it does not prove those blocks assemble into a bootable system, that an application can replay its logs into them, or that every link in a long incremental chain is present. Many products also verify a synthesized or latest point, while a real incident needs a specific older point with a longer and more fragile chain. Application consistency is another gap: a crash consistent image can verify perfectly and still leave a database needing recovery that fails. Finally, verification often runs with generous time and local repository access, while a real restore runs under pressure to a different target with different drivers, network paths, and permissions. The pass was real - it just tested less than the restore needed.

Variants and edge cases

checksum passes, database will not open

The blocks are intact but the database needs log replay, a missing wallet or key file, or a version match that verification never attempted. Test restores for databases must include opening the database and running one read query, ideally the same query the application health check uses.

newest point verifies, older point fails

Long chains accumulate risk: one damaged incremental invalidates every later point that depends on it. Verification that only touches the newest synthesized point can miss a broken middle link if synthesis used a cached result. Force a chain read from the full forward and quarantine points after the break.

boot test passes in sandbox, fails on real hardware

Sandbox boot tests use generic virtual hardware. Real targets add storage controller, firmware mode, and secure boot differences. Keep a target compatibility check in the restore runbook and record the sandbox configuration so the difference is visible. Edge cases: verification can pass against a cached or replicated copy while the primary copy is damaged, so record which repository copy each test read. Time skew matters too: a point verified last month can degrade later through bit rot or repository migration, which is why per point results need dates. The operational fix is cultural as much as technical - report verification coverage, meaning what layer and which points were tested, next to the pass mark, so a green dashboard never silently means checksums only on yesterday's point.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O-BuPDr5z-uQasrCUYGv0w

Resolved from the public thread: https://vectle.com/posts/pst_O-BuPDr5z-uQasrCUYGv0w

Thread query: backup verification passes but actual restore fails

TL;DR: Do not trust the passing verification until you know exactly what it tested: most verification only checks block checksums or mounts a file system, it does not boot the system, start the application, or replay the full chain a real restore needs. Reproduce the real restore in isolation, compare it step for step with the verification path, and widen verification to cover the failing layer.