backup verification passes but actual restore fails

TL;DR

Do not trust the passing verification until you know exactly what it tested: most verification only checks block checksums or mounts a file system, it does not boot the system, start the application, or replay the full chain a real restore needs. Reproduce the real restore in isolation, compare it step for step with the verification path, and widen verification to cover the failing layer.

backup verification passes but actual restore fails

Use this when

• Verification or health checks pass, but a real restore fails, boots badly, or returns corrupt data.
• Verification tests the newest point only while the failed restore used an older or longer chain.
• A file level verification passes for a system that must boot or run a database.

Not for this skill when

• Verification itself reports checksum or repository errors. Fix the repository fault first.
• The restore failed for a one off target hardware or driver reason. Use the dissimilar hardware skill.

Steps

1. Write down exactly what the passing verification does in your product: checksum only, file system mount, boot test, or application test, and which restore point and chain length it used. Verify: the verification scope is stated in one sentence and matches its log, not its job name
2. Repeat the failed real restore into an isolated target, capturing where it stops: chain assembly, data read, boot, service start, or application open. Verify: the failure layer is named with the exact error and timestamp from the restore log
3. Restore the exact point verification tested and the exact point that failed in production, side by side, and compare chain length, included disks, and application consistency settings. Verify: any difference in point, chain, or options between the two runs is listed
4. Run a full chain test: apply the full plus every incremental up to the target point instead of letting verification synthesize or shortcut the chain. Verify: every chain link is read during the test and a damaged middle link, if present, is identified by date
5. Add the missing layer to verification: a boot test for system images, a database open and query test for database backups, and a sample file hash comparison for file backups. Verify: the widened test fails when you deliberately point it at the known bad restore point
6. Schedule the widened verification across the retention window, not only on the newest point, and keep its results per point so a failing point is quarantined. Verify: older points appear in the report with individual pass or fail results

Why it happens

Verification and restore exercise different code paths. A checksum verification proves stored blocks match the hashes recorded when they were written; it does not prove those blocks assemble into a bootable system, that an application can replay its logs into them, or that every link in a long incremental chain is present. Many products also verify a synthesized or latest point, while a real incident needs a specific older point with a longer and more fragile chain. Application consistency is another gap: a crash consistent image can verify perfectly and still leave a database needing recovery that fails. Finally, verification often runs with generous time and local repository access, while a real restore runs under pressure to a different target with different drivers, network paths, and permissions. The pass was real - it just tested less than the restore needed.

Variants and edge cases

checksum passes, database will not open

The blocks are intact but the database needs log replay, a missing wallet or key file, or a version match that verification never attempted. Test restores for databases must include opening the database and running one read query, ideally the same query the application health check uses.

newest point verifies, older point fails

Long chains accumulate risk: one damaged incremental invalidates every later point that depends on it. Verification that only touches the newest synthesized point can miss a broken middle link if synthesis used a cached result. Force a chain read from the full forward and quarantine points after the break.

boot test passes in sandbox, fails on real hardware

Sandbox boot tests use generic virtual hardware. Real targets add storage controller, firmware mode, and secure boot differences. Keep a target compatibility check in the restore runbook and record the sandbox configuration so the difference is visible. Edge cases: verification can pass against a cached or replicated copy while the primary copy is damaged, so record which repository copy each test read. Time skew matters too: a point verified last month can degrade later through bit rot or repository migration, which is why per point results need dates. The operational fix is cultural as much as technical - report verification coverage, meaning what layer and which points were tested, next to the pass mark, so a green dashboard never silently means checksums only on yesterday's point.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O-BuPDr5z-uQasrCUYGv0w