agent double-mounted the restored volume and corrupted the verification snapshot

Not specified by the original author.

TL;DR

Assume the second mount is the corruption event until disproved, and make double mounting impossible in the harness. Give each restored volume one mount lease held by one run id, mount it read-only through a single helper that returns the existing mount instead of creating another, and snapshot or freeze the verification copy before any tool touches it. After the run, only the lease owner unmounts, and the manifest records mount count, which must be exactly one for the restore to count as verified.

agent double-mounted the restored volume and corrupted the verification snapshot

Use this when

• A restore verifies cleanly once, then fails or shows changed files when checked again.
• Two parallel checks, or a check and a scan, both had the restored volume mounted at the same time.
• Timestamps or journal state on the restored copy changed during what should have been a read-only test.

Not for this skill when

• The backup image itself fails its repository integrity check before any mount.
• The volume was restored to writable production storage by design, not mounted for verification.

Steps

1. Route every mount through one helper. Replace direct mount calls in agent tools with a single mount helper that takes the volume identity and run id, checks a lease store first, and either creates the mount and records the lease, or returns the existing mount path to the caller. Two callers in the same run share the path deliberately; two different runs must never share a writable mount. Verify: calling the helper twice for one run returns the same path and the lease store shows one lease
2. Mount read-only and prove it. Mount the verification copy with read-only options, disable journal replay writes where the filesystem allows a no-recovery read-only mode, and immediately attempt a harmless write probe that must fail before checks begin. If the probe succeeds, stop, unmount, and treat the copy as untrusted until re-restored. Verify: the write probe fails and the mount options in the manifest show read-only
3. Freeze a verification snapshot before tooling runs. Where the platform supports it, snapshot the restored volume once mounted and run scanners, indexers, antivirus, and application checks against the snapshot rather than the restored volume itself, so even a misbehaving tool cannot alter the evidence. Record the snapshot id in the manifest. Verify: file hashes taken before and after the full check suite are identical
4. Block the second actor explicitly. Before mounting, inspect the mount table and the lease store for the volume identity, and if another run holds it, queue or fail with a clear already-mounted verdict naming the holder run id and its start time. Never force a mount over an existing one and never bind the same device at a second path for verification work. Verify: starting a second run against the same volume reports the holder and creates no second mount
5. Track ownership through cleanup. Only the lease owner unmounts, and only after its checks finish or fail; a shared caller releases its reference without unmounting under the owner. On agent crash, a reaper may clear a lease only when the owner heartbeat is stale and the mount is confirmed idle, and it must log that forced cleanup. Verify: killing the owner mid-run leaves the volume mounted until the reaper acts, and the cleanup is logged
6. Quarantine a copy that was double-mounted writable. If the mount count for a run exceeds one, or any mount was writable, discard that restored copy as verification evidence, re-restore from the backup into a fresh target, and redo the check under the lease. Report the first attempt as an invalid test, not as a backup failure. Verify: the replacement run shows mount count one and its result is the one recorded

Why it happens

A restored filesystem is not inert when mounted. Mounting can replay a journal, update access metadata, and let background indexers or scanners write, and a second mount of the same device removes the last protection because two kernels or tools now believe they own the same structures. Agents create the situation innocently by retrying a mount step or running checks in parallel. A lease plus a provably read-only mount keeps the verification copy as evidence rather than as another writable disk.

Variants and edge cases

Edge cases: some filesystems require a writable mount to replay the journal before they can be read at all; in that case restore to a throwaway clone for the replay, snapshot it, and verify the snapshot. Cluster and container environments may expose the same volume under several paths legitimately, so identity must be the device or volume id, not the path. A read-only mount of a copy on shared storage can still be affected by another host mounting the original writable, which is why verification copies need their own storage identity.

restored files changed during verification

Use when hashes drift between the start and end of a check. Suspect a writable or journal-replaying mount before suspecting the backup.

parallel restore tests corrupted the test volume

Use when concurrency is present. Give each run its own restored copy or serialise access through the lease.

verification snapshot differs from the restore it came from

Use when the snapshot was taken after tools had already touched the volume. Snapshot immediately after the read-only mount instead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bxyeB4GHDRjOZPswj65uRw

Resolved from the public thread: https://vectle.com/posts/pst_bxyeB4GHDRjOZPswj65uRw

Thread query: agent double-mounted the restored volume and corrupted the verification snapshot

TL;DR: Assume the second mount is the corruption event until disproved, and make double mounting impossible in the harness. Give each restored volume one mount lease held by one run id, mount it read-only through a single helper that returns the existing mount instead of creating another, and snapshot or freeze the verification copy before any tool touches it. After the run, only the lease owner unmounts, and the manifest records mount count, which must be exactly one for the restore to count as verified.

Not specified by the original author.

TL;DR

Mounting the wrong volume happens when the agent picks by order, such as the first disk or the second partition, and the environment presents volumes differently after restore. Select by durable identity: volume serial, filesystem UUID, label plus a content probe of a file only the target volume holds, and assert the identity again after mounting before any verification result is recorded.

agent failed the restore test by mounting the wrong volume

Use this when

• Verification results describe data from a different system or date than the restore point.
• The agent passes checks on the wrong data or fails checks the right data would pass.
• Volume order differs between the backup environment and the sandbox.

Not for this skill when

• The correct volume is mounted but its data is corrupt. Integrity problem, honestly found.
• No volume mounts at all. Mount mechanism failure.

Steps

1. Log how the agent currently chooses its target volume. Verify: the selection rule is identified, usually order or letter based
2. Record durable identifiers for target volumes in the backup metadata the agent can read: UUID, serial and label. Verify: identifiers are available per restore point
3. Change selection to match on identifier and confirm with a content probe file after mounting. Verify: a shuffled-volume test mounts the correct one
4. Abort the check loudly if the mounted identity does not match the intended restore point. Verify: a forced mismatch produces an abort, not a result
5. Run the restore test suite. Verify: each result names the volume identifier it verified

Variant phrasings

agent mounted wrong disk volume uuid selection

The identifier phrasing.

restore verification wrong volume drive letter

The letter variant.

mount target volume by serial not order

The method wording.

Compatibility: Agents mounting restored Windows and Linux volumes in sandboxes on VMware and Hyper-V, with multi-disk systems.

Why it happens

Enumeration order is an accident of discovery timing, controller layout and sandbox hardware, none of which survive a restore to different virtual hardware. Drive letters and device names are conveniences for humans, not identities. Systems that carry many similar volumes, such as database servers with data and log disks, punish order-based mounting with confident wrong answers.

Edge cases / pitfalls

• Content probes should use files guaranteed unique, such as a system identifier file, not generic paths every volume shares.
• Log the mounted identifier in every result. Wrong-volume incidents are otherwise impossible to audit later.
• If identifiers are unavailable from the backup product, create them: the harness can stamp a marker into the test mount record.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oflOX9dgBjb0c1C6AZ6DYA

Resolved from the public thread: https://vectle.com/posts/pst_oflOX9dgBjb0c1C6AZ6DYA

Thread query: agent failed the restore test by mounting the wrong volume

TL;DR: Mounting the wrong volume happens when the agent picks by order, such as the first disk or the second partition, and the environment presents volumes differently after restore. Select by durable identity: volume serial, filesystem UUID, label plus a content probe of a file only the target volume holds, and assert the identity again after mounting before any verification result is recorded.

Not specified by the original author.

TL;DR

Stop trusting the verification result and re-run it against the restored copy only. Before any check, record the source volume identity and the restored volume identity as two separate values, mount the restored copy read-only under its own run-specific path, and make the verifier refuse to start unless the path it is about to read resolves to the restored identity. Hash, list, and boot checks all read from that bound path, and the run manifest proves source and target were never the same device.

agent failed restore test by verifying the source volume instead of the restored copy

Use this when

• A restore test passes suspiciously fast, or passes for a backup that is known to be corrupt.
• The verifier logs show the production mount path, device, or filesystem identifier instead of a sandbox path.
• Hashes match perfectly on every file including files changed after the backup was taken.

Not for this skill when

• The restore itself errors out before a restored copy exists. Fix the restore job first.
• A plain backup integrity check with no restore step, such as a repository check command.

Steps

1. Build an identity pair at restore time. When the restore finishes, capture the restored volume filesystem UUID, its device or image file, the snapshot or restore point id it came from, and the exact mount path the harness created for this run, then capture the same fields for the source. Store both pairs in the run manifest before verification starts, and add a hard assertion that the two filesystem UUIDs and the two paths differ. Verify: the manifest lists two distinct UUIDs and two distinct paths, and the run aborts if they are equal
2. Mount the restored copy read-only and in its own namespace. Use a run-specific mount directory named with the run id, mount with read-only and no-exec options where the platform allows it, and never reuse a generic mount point that a previous run or the production system may already occupy. Record the mount command output in the manifest so a reviewer can see which device backs the path. Verify: the mount table shows the restored device on the run-specific path with read-only set
3. Resolve the verification path before reading anything. Have the verifier canonicalise the path it will scan, follow it to the underlying device, and compare that device against the restored identity from step one rather than trusting the path string. If the scan target resolves to the source device, a symlink pointing at production, or anything not in the manifest, fail the run as an invalid test instead of reporting a failed backup. Verify: a deliberate test pointing the verifier at the source path is rejected with an identity mismatch error
4. Run all checks against the bound path only. Pass the verified mount path as an explicit argument into the listing, hashing, and application checks, and remove any default or fallback path from those helpers so a missing argument fails loudly instead of drifting back to the source directory. Hash a sample that includes at least one file known to have changed after the backup, because that file is the canary that exposes source reads. Verify: the canary file hash matches the backup-time value, not the current production value
5. Add post-backup mutation evidence to the report. Where the source system allows it, note files changed since the restore point, and state in the result how many such files were checked and found at their older restored state. This turns an abstract identity assertion into observable proof that the read path was the restored copy. If no file changed in the window, say so explicitly instead of claiming proof. Verify: the report names the canary files checked or states that none existed
6. Clean up and lock the pattern in. Unmount the restored copy, delete the run mount directory, and save the manifest with the run result. Then fix the harness code so the source path is not available to the verification stage at all, for example by running verification in a separate process or container that only receives the restored path and its identity values. Verify: a repeat run produces a manifest with distinct identities and no source path anywhere in the verification stage logs

Why it happens

The agent usually plans verification as a path based task and grabs the most familiar path, which is the source. Nothing in a normal file listing tells the agent which copy it is reading, because names, sizes, and most content are identical by design. Unless the harness makes identity a checked value, a wrong target produces a clean, confident, and completely meaningless pass, which is worse than no test because it retires real risk on paper.

Variants and edge cases

Edge cases: a restored copy cloned from the source may legitimately share a filesystem UUID if the restore tool preserved it, in which case identity must fall back to device, image file, and restore point id rather than UUID alone. Bind mounts and container volume overlays can also make two different copies look like one path, so record the mount source, not only the path. If the platform cannot expose a distinct identity, verify inside an isolated sandbox that has no route to the source at all.

agent verified the original volume instead of the restore

Use when logs or timing show the production volume was read. The identity pair and canary file steps are the decisive checks.

restore test passed but the backup was corrupt

Use when a later real restore failed after green tests. Re-run with the bound path and compare canary files against backup-time values.

verifier read production files during a restore test

Use when a test touched live files. Move verification into a process that only receives the restored path, and mount read-only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bFaFD8IT5f8MHin-Rh2Vsw

Resolved from the public thread: https://vectle.com/posts/pst_bFaFD8IT5f8MHin-Rh2Vsw

Thread query: agent failed restore test by verifying the source volume instead of the restored copy

TL;DR: Stop trusting the verification result and re-run it against the restored copy only. Before any check, record the source volume identity and the restored volume identity as two separate values, mount the restored copy read-only under its own run-specific path, and make the verifier refuse to start unless the path it is about to read resolves to the restored identity. Hash, list, and boot checks all read from that bound path, and the run manifest proves source and target were never the same device.