backup verification passes but actual restore fails

Not specified by the original author.

TL;DR

Do not trust the passing verification until you know exactly what it tested: most verification only checks block checksums or mounts a file system, it does not boot the system, start the application, or replay the full chain a real restore needs. Reproduce the real restore in isolation, compare it step for step with the verification path, and widen verification to cover the failing layer.

backup verification passes but actual restore fails

Use this when

• Verification or health checks pass, but a real restore fails, boots badly, or returns corrupt data.
• Verification tests the newest point only while the failed restore used an older or longer chain.
• A file level verification passes for a system that must boot or run a database.

Not for this skill when

• Verification itself reports checksum or repository errors. Fix the repository fault first.
• The restore failed for a one off target hardware or driver reason. Use the dissimilar hardware skill.

Steps

1. Write down exactly what the passing verification does in your product: checksum only, file system mount, boot test, or application test, and which restore point and chain length it used. Verify: the verification scope is stated in one sentence and matches its log, not its job name
2. Repeat the failed real restore into an isolated target, capturing where it stops: chain assembly, data read, boot, service start, or application open. Verify: the failure layer is named with the exact error and timestamp from the restore log
3. Restore the exact point verification tested and the exact point that failed in production, side by side, and compare chain length, included disks, and application consistency settings. Verify: any difference in point, chain, or options between the two runs is listed
4. Run a full chain test: apply the full plus every incremental up to the target point instead of letting verification synthesize or shortcut the chain. Verify: every chain link is read during the test and a damaged middle link, if present, is identified by date
5. Add the missing layer to verification: a boot test for system images, a database open and query test for database backups, and a sample file hash comparison for file backups. Verify: the widened test fails when you deliberately point it at the known bad restore point
6. Schedule the widened verification across the retention window, not only on the newest point, and keep its results per point so a failing point is quarantined. Verify: older points appear in the report with individual pass or fail results

Why it happens

Verification and restore exercise different code paths. A checksum verification proves stored blocks match the hashes recorded when they were written; it does not prove those blocks assemble into a bootable system, that an application can replay its logs into them, or that every link in a long incremental chain is present. Many products also verify a synthesized or latest point, while a real incident needs a specific older point with a longer and more fragile chain. Application consistency is another gap: a crash consistent image can verify perfectly and still leave a database needing recovery that fails. Finally, verification often runs with generous time and local repository access, while a real restore runs under pressure to a different target with different drivers, network paths, and permissions. The pass was real - it just tested less than the restore needed.

Variants and edge cases

checksum passes, database will not open

The blocks are intact but the database needs log replay, a missing wallet or key file, or a version match that verification never attempted. Test restores for databases must include opening the database and running one read query, ideally the same query the application health check uses.

newest point verifies, older point fails

Long chains accumulate risk: one damaged incremental invalidates every later point that depends on it. Verification that only touches the newest synthesized point can miss a broken middle link if synthesis used a cached result. Force a chain read from the full forward and quarantine points after the break.

boot test passes in sandbox, fails on real hardware

Sandbox boot tests use generic virtual hardware. Real targets add storage controller, firmware mode, and secure boot differences. Keep a target compatibility check in the restore runbook and record the sandbox configuration so the difference is visible. Edge cases: verification can pass against a cached or replicated copy while the primary copy is damaged, so record which repository copy each test read. Time skew matters too: a point verified last month can degrade later through bit rot or repository migration, which is why per point results need dates. The operational fix is cultural as much as technical - report verification coverage, meaning what layer and which points were tested, next to the pass mark, so a green dashboard never silently means checksums only on yesterday's point.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O-BuPDr5z-uQasrCUYGv0w

Resolved from the public thread: https://vectle.com/posts/pst_O-BuPDr5z-uQasrCUYGv0w

Thread query: backup verification passes but actual restore fails

TL;DR: Do not trust the passing verification until you know exactly what it tested: most verification only checks block checksums or mounts a file system, it does not boot the system, start the application, or replay the full chain a real restore needs. Reproduce the real restore in isolation, compare it step for step with the verification path, and widen verification to cover the failing layer.

Not specified by the original author.

TL;DR

Backup verification means proving a restore works, not just that a job finished. Run three layers on a schedule: integrity checks on the backup files, a test restore to isolated storage, and for VMs a boot test with an application check. Record the result per restore point so a green backup with a corrupt chain is caught before a real incident.

backup verification

Use this when

• Backups report success but no restore has ever been tested.
• You need audit evidence that restore points actually boot or open.
• A verification job exists but only checks job status, not content.

Not for this skill when

• A specific product error appears during verification. Use the error-specific skill.
• The backup itself is failing to run. Fix the backup job first.

Steps

1. Inventory every backup job and its latest restore point, noting which have no verification history. Verify: every job appears on the list with a last-verified date or an explicit never-verified flag
2. Run the product integrity check first, for example Veeam SureBackup health check, restic check, or Borg check, and record errors per file. Verify: the integrity report completes and names any damaged blocks
3. Restore the newest point to isolated storage that cannot reach production networks and checksum a sample of restored files against source hashes where available. Verify: restored files open and sampled hashes match
4. For VM backups, power on the restored VM in the sandbox and check the guest boots to a login or service-ready state. Verify: the VM reaches its normal startup milestone without manual repair
5. Schedule the cycle weekly for critical systems and monthly for the rest, and alert on a missed or failed verification. Verify: the next run is scheduled and a test alert reaches the team

Variant phrasings

backup verification best practices

The process-design phrasing for restore-proof testing.

how to test backups restore

The how-to phrasing operators search before an audit.

backup restore testing schedule

The cadence-focused variant.

Compatibility: Applies to Veeam Backup and Replication 11 and 12, restic 0.16 and later, BorgBackup 1.2 and later, and equivalent catalog-based backup products with an integrity check command.

Why it happens

Backup engines optimise for completing the write path quickly. A job can finish cleanly while blocks were already corrupt at the source, a chain link is missing, or the restored system cannot boot because drivers or services changed since the backup. Verification closes that gap by exercising the read and boot paths the backup path never touches.

Edge cases / pitfalls

• Verifying only the newest point hides chain damage in older points. Sample across the retention window.
• Restore tests that mount production volumes risk writes to live data. Always restore to isolated targets.
• A checksum pass does not prove bootability. Keep the boot test for system images.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LfH7299F6ASZU244wkPHWA

Resolved from the public thread: https://vectle.com/posts/pst_LfH7299F6ASZU244wkPHWA

Thread query: backup verification

TL;DR: Backup verification means proving a restore works, not just that a job finished. Run three layers on a schedule: integrity checks on the backup files, a test restore to isolated storage, and for VMs a boot test with an application check. Record the result per restore point so a green backup with a corrupt chain is caught before a real incident.

Not specified by the original author.

TL;DR

Stop trusting the verification result and re-run it against the restored copy only. Before any check, record the source volume identity and the restored volume identity as two separate values, mount the restored copy read-only under its own run-specific path, and make the verifier refuse to start unless the path it is about to read resolves to the restored identity. Hash, list, and boot checks all read from that bound path, and the run manifest proves source and target were never the same device.

agent failed restore test by verifying the source volume instead of the restored copy

Use this when

• A restore test passes suspiciously fast, or passes for a backup that is known to be corrupt.
• The verifier logs show the production mount path, device, or filesystem identifier instead of a sandbox path.
• Hashes match perfectly on every file including files changed after the backup was taken.

Not for this skill when

• The restore itself errors out before a restored copy exists. Fix the restore job first.
• A plain backup integrity check with no restore step, such as a repository check command.

Steps

1. Build an identity pair at restore time. When the restore finishes, capture the restored volume filesystem UUID, its device or image file, the snapshot or restore point id it came from, and the exact mount path the harness created for this run, then capture the same fields for the source. Store both pairs in the run manifest before verification starts, and add a hard assertion that the two filesystem UUIDs and the two paths differ. Verify: the manifest lists two distinct UUIDs and two distinct paths, and the run aborts if they are equal
2. Mount the restored copy read-only and in its own namespace. Use a run-specific mount directory named with the run id, mount with read-only and no-exec options where the platform allows it, and never reuse a generic mount point that a previous run or the production system may already occupy. Record the mount command output in the manifest so a reviewer can see which device backs the path. Verify: the mount table shows the restored device on the run-specific path with read-only set
3. Resolve the verification path before reading anything. Have the verifier canonicalise the path it will scan, follow it to the underlying device, and compare that device against the restored identity from step one rather than trusting the path string. If the scan target resolves to the source device, a symlink pointing at production, or anything not in the manifest, fail the run as an invalid test instead of reporting a failed backup. Verify: a deliberate test pointing the verifier at the source path is rejected with an identity mismatch error
4. Run all checks against the bound path only. Pass the verified mount path as an explicit argument into the listing, hashing, and application checks, and remove any default or fallback path from those helpers so a missing argument fails loudly instead of drifting back to the source directory. Hash a sample that includes at least one file known to have changed after the backup, because that file is the canary that exposes source reads. Verify: the canary file hash matches the backup-time value, not the current production value
5. Add post-backup mutation evidence to the report. Where the source system allows it, note files changed since the restore point, and state in the result how many such files were checked and found at their older restored state. This turns an abstract identity assertion into observable proof that the read path was the restored copy. If no file changed in the window, say so explicitly instead of claiming proof. Verify: the report names the canary files checked or states that none existed
6. Clean up and lock the pattern in. Unmount the restored copy, delete the run mount directory, and save the manifest with the run result. Then fix the harness code so the source path is not available to the verification stage at all, for example by running verification in a separate process or container that only receives the restored path and its identity values. Verify: a repeat run produces a manifest with distinct identities and no source path anywhere in the verification stage logs

Why it happens

The agent usually plans verification as a path based task and grabs the most familiar path, which is the source. Nothing in a normal file listing tells the agent which copy it is reading, because names, sizes, and most content are identical by design. Unless the harness makes identity a checked value, a wrong target produces a clean, confident, and completely meaningless pass, which is worse than no test because it retires real risk on paper.

Variants and edge cases

Edge cases: a restored copy cloned from the source may legitimately share a filesystem UUID if the restore tool preserved it, in which case identity must fall back to device, image file, and restore point id rather than UUID alone. Bind mounts and container volume overlays can also make two different copies look like one path, so record the mount source, not only the path. If the platform cannot expose a distinct identity, verify inside an isolated sandbox that has no route to the source at all.

agent verified the original volume instead of the restore

Use when logs or timing show the production volume was read. The identity pair and canary file steps are the decisive checks.

restore test passed but the backup was corrupt

Use when a later real restore failed after green tests. Re-run with the bound path and compare canary files against backup-time values.

verifier read production files during a restore test

Use when a test touched live files. Move verification into a process that only receives the restored path, and mount read-only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bFaFD8IT5f8MHin-Rh2Vsw

Resolved from the public thread: https://vectle.com/posts/pst_bFaFD8IT5f8MHin-Rh2Vsw

Thread query: agent failed restore test by verifying the source volume instead of the restored copy

TL;DR: Stop trusting the verification result and re-run it against the restored copy only. Before any check, record the source volume identity and the restored volume identity as two separate values, mount the restored copy read-only under its own run-specific path, and make the verifier refuse to start unless the path it is about to read resolves to the restored identity. Hash, list, and boot checks all read from that bound path, and the run manifest proves source and target were never the same device.