agent failed restore test by verifying the source volume instead of the restored copy
Fixes restore verification that silently checks the source volume instead of the restored copy, by binding every check to the restored volume identity and path. Use when an agent reports a verified restore that never touched the backup. Not for backups that failed to run in the first place.
TL;DR
Stop trusting the verification result and re-run it against the restored copy only. Before any check, record the source volume identity and the restored volume identity as two separate values, mount the restored copy read-only under its own run-specific path, and make the verifier refuse to start unless the path it is about to read resolves to the restored identity. Hash, list, and boot checks all read from that bound path, and the run manifest proves source and target were never the same device.
agent failed restore test by verifying the source volume instead of the restored copyUse this when
- A restore test passes suspiciously fast, or passes for a backup that is known to be corrupt.
- The verifier logs show the production mount path, device, or filesystem identifier instead of a sandbox path.
- Hashes match perfectly on every file including files changed after the backup was taken.
Not for this skill when
- The restore itself errors out before a restored copy exists. Fix the restore job first.
- A plain backup integrity check with no restore step, such as a repository check command.
Steps
- Build an identity pair at restore time. When the restore finishes, capture the restored volume filesystem UUID, its device or image file, the snapshot or restore point id it came from, and the exact mount path the harness created for this run, then capture the same fields for the source. Store both pairs in the run manifest before verification starts, and add a hard assertion that the two filesystem UUIDs and the two paths differ. Verify: the manifest lists two distinct UUIDs and two distinct paths, and the run aborts if they are equal
- Mount the restored copy read-only and in its own namespace. Use a run-specific mount directory named with the run id, mount with read-only and no-exec options where the platform allows it, and never reuse a generic mount point that a previous run or the production system may already occupy. Record the mount command output in the manifest so a reviewer can see which device backs the path. Verify: the mount table shows the restored device on the run-specific path with read-only set
- Resolve the verification path before reading anything. Have the verifier canonicalise the path it will scan, follow it to the underlying device, and compare that device against the restored identity from step one rather than trusting the path string. If the scan target resolves to the source device, a symlink pointing at production, or anything not in the manifest, fail the run as an invalid test instead of reporting a failed backup. Verify: a deliberate test pointing the verifier at the source path is rejected with an identity mismatch error
- Run all checks against the bound path only. Pass the verified mount path as an explicit argument into the listing, hashing, and application checks, and remove any default or fallback path from those helpers so a missing argument fails loudly instead of drifting back to the source directory. Hash a sample that includes at least one file known to have changed after the backup, because that file is the canary that exposes source reads. Verify: the canary file hash matches the backup-time value, not the current production value
- Add post-backup mutation evidence to the report. Where the source system allows it, note files changed since the restore point, and state in the result how many such files were checked and found at their older restored state. This turns an abstract identity assertion into observable proof that the read path was the restored copy. If no file changed in the window, say so explicitly instead of claiming proof. Verify: the report names the canary files checked or states that none existed
- Clean up and lock the pattern in. Unmount the restored copy, delete the run mount directory, and save the manifest with the run result. Then fix the harness code so the source path is not available to the verification stage at all, for example by running verification in a separate process or container that only receives the restored path and its identity values. Verify: a repeat run produces a manifest with distinct identities and no source path anywhere in the verification stage logs
Why it happens
The agent usually plans verification as a path based task and grabs the most familiar path, which is the source. Nothing in a normal file listing tells the agent which copy it is reading, because names, sizes, and most content are identical by design. Unless the harness makes identity a checked value, a wrong target produces a clean, confident, and completely meaningless pass, which is worse than no test because it retires real risk on paper.
Variants and edge cases
Edge cases: a restored copy cloned from the source may legitimately share a filesystem UUID if the restore tool preserved it, in which case identity must fall back to device, image file, and restore point id rather than UUID alone. Bind mounts and container volume overlays can also make two different copies look like one path, so record the mount source, not only the path. If the platform cannot expose a distinct identity, verify inside an isolated sandbox that has no route to the source at all.
agent verified the original volume instead of the restore
Use when logs or timing show the production volume was read. The identity pair and canary file steps are the decisive checks.
restore test passed but the backup was corrupt
Use when a later real restore failed after green tests. Re-run with the bound path and compare canary files against backup-time values.
verifier read production files during a restore test
Use when a test touched live files. Move verification into a process that only receives the restored path, and mount read-only.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_bFaFD8IT5f8MHin-Rh2Vsw