agent double-mounted the restored volume and corrupted the verification snapshot

TL;DR

Assume the second mount is the corruption event until disproved, and make double mounting impossible in the harness. Give each restored volume one mount lease held by one run id, mount it read-only through a single helper that returns the existing mount instead of creating another, and snapshot or freeze the verification copy before any tool touches it. After the run, only the lease owner unmounts, and the manifest records mount count, which must be exactly one for the restore to count as verified.

agent double-mounted the restored volume and corrupted the verification snapshot

Use this when

• A restore verifies cleanly once, then fails or shows changed files when checked again.
• Two parallel checks, or a check and a scan, both had the restored volume mounted at the same time.
• Timestamps or journal state on the restored copy changed during what should have been a read-only test.

Not for this skill when

• The backup image itself fails its repository integrity check before any mount.
• The volume was restored to writable production storage by design, not mounted for verification.

Steps

1. Route every mount through one helper. Replace direct mount calls in agent tools with a single mount helper that takes the volume identity and run id, checks a lease store first, and either creates the mount and records the lease, or returns the existing mount path to the caller. Two callers in the same run share the path deliberately; two different runs must never share a writable mount. Verify: calling the helper twice for one run returns the same path and the lease store shows one lease
2. Mount read-only and prove it. Mount the verification copy with read-only options, disable journal replay writes where the filesystem allows a no-recovery read-only mode, and immediately attempt a harmless write probe that must fail before checks begin. If the probe succeeds, stop, unmount, and treat the copy as untrusted until re-restored. Verify: the write probe fails and the mount options in the manifest show read-only
3. Freeze a verification snapshot before tooling runs. Where the platform supports it, snapshot the restored volume once mounted and run scanners, indexers, antivirus, and application checks against the snapshot rather than the restored volume itself, so even a misbehaving tool cannot alter the evidence. Record the snapshot id in the manifest. Verify: file hashes taken before and after the full check suite are identical
4. Block the second actor explicitly. Before mounting, inspect the mount table and the lease store for the volume identity, and if another run holds it, queue or fail with a clear already-mounted verdict naming the holder run id and its start time. Never force a mount over an existing one and never bind the same device at a second path for verification work. Verify: starting a second run against the same volume reports the holder and creates no second mount
5. Track ownership through cleanup. Only the lease owner unmounts, and only after its checks finish or fail; a shared caller releases its reference without unmounting under the owner. On agent crash, a reaper may clear a lease only when the owner heartbeat is stale and the mount is confirmed idle, and it must log that forced cleanup. Verify: killing the owner mid-run leaves the volume mounted until the reaper acts, and the cleanup is logged
6. Quarantine a copy that was double-mounted writable. If the mount count for a run exceeds one, or any mount was writable, discard that restored copy as verification evidence, re-restore from the backup into a fresh target, and redo the check under the lease. Report the first attempt as an invalid test, not as a backup failure. Verify: the replacement run shows mount count one and its result is the one recorded

Why it happens

A restored filesystem is not inert when mounted. Mounting can replay a journal, update access metadata, and let background indexers or scanners write, and a second mount of the same device removes the last protection because two kernels or tools now believe they own the same structures. Agents create the situation innocently by retrying a mount step or running checks in parallel. A lease plus a provably read-only mount keeps the verification copy as evidence rather than as another writable disk.

Variants and edge cases

Edge cases: some filesystems require a writable mount to replay the journal before they can be read at all; in that case restore to a throwaway clone for the replay, snapshot it, and verify the snapshot. Cluster and container environments may expose the same volume under several paths legitimately, so identity must be the device or volume id, not the path. A read-only mount of a copy on shared storage can still be affected by another host mounting the original writable, which is why verification copies need their own storage identity.

restored files changed during verification

Use when hashes drift between the start and end of a check. Suspect a writable or journal-replaying mount before suspecting the backup.

parallel restore tests corrupted the test volume

Use when concurrency is present. Give each run its own restored copy or serialise access through the lease.

verification snapshot differs from the restore it came from

Use when the snapshot was taken after tools had already touched the volume. Snapshot immediately after the read-only mount instead.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bxyeB4GHDRjOZPswj65uRw