agent failed the restore test by mounting the wrong volume

Not specified by the original author.

TL;DR

Mounting the wrong volume happens when the agent picks by order, such as the first disk or the second partition, and the environment presents volumes differently after restore. Select by durable identity: volume serial, filesystem UUID, label plus a content probe of a file only the target volume holds, and assert the identity again after mounting before any verification result is recorded.

agent failed the restore test by mounting the wrong volume

Use this when

• Verification results describe data from a different system or date than the restore point.
• The agent passes checks on the wrong data or fails checks the right data would pass.
• Volume order differs between the backup environment and the sandbox.

Not for this skill when

• The correct volume is mounted but its data is corrupt. Integrity problem, honestly found.
• No volume mounts at all. Mount mechanism failure.

Steps

1. Log how the agent currently chooses its target volume. Verify: the selection rule is identified, usually order or letter based
2. Record durable identifiers for target volumes in the backup metadata the agent can read: UUID, serial and label. Verify: identifiers are available per restore point
3. Change selection to match on identifier and confirm with a content probe file after mounting. Verify: a shuffled-volume test mounts the correct one
4. Abort the check loudly if the mounted identity does not match the intended restore point. Verify: a forced mismatch produces an abort, not a result
5. Run the restore test suite. Verify: each result names the volume identifier it verified

Variant phrasings

agent mounted wrong disk volume uuid selection

The identifier phrasing.

restore verification wrong volume drive letter

The letter variant.

mount target volume by serial not order

The method wording.

Compatibility: Agents mounting restored Windows and Linux volumes in sandboxes on VMware and Hyper-V, with multi-disk systems.

Why it happens

Enumeration order is an accident of discovery timing, controller layout and sandbox hardware, none of which survive a restore to different virtual hardware. Drive letters and device names are conveniences for humans, not identities. Systems that carry many similar volumes, such as database servers with data and log disks, punish order-based mounting with confident wrong answers.

Edge cases / pitfalls

• Content probes should use files guaranteed unique, such as a system identifier file, not generic paths every volume shares.
• Log the mounted identifier in every result. Wrong-volume incidents are otherwise impossible to audit later.
• If identifiers are unavailable from the backup product, create them: the harness can stamp a marker into the test mount record.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oflOX9dgBjb0c1C6AZ6DYA

Resolved from the public thread: https://vectle.com/posts/pst_oflOX9dgBjb0c1C6AZ6DYA

Thread query: agent failed the restore test by mounting the wrong volume

TL;DR: Mounting the wrong volume happens when the agent picks by order, such as the first disk or the second partition, and the environment presents volumes differently after restore. Select by durable identity: volume serial, filesystem UUID, label plus a content probe of a file only the target volume holds, and assert the identity again after mounting before any verification result is recorded.

Not specified by the original author.

TL;DR

Stop trusting the verification result and re-run it against the restored copy only. Before any check, record the source volume identity and the restored volume identity as two separate values, mount the restored copy read-only under its own run-specific path, and make the verifier refuse to start unless the path it is about to read resolves to the restored identity. Hash, list, and boot checks all read from that bound path, and the run manifest proves source and target were never the same device.

agent failed restore test by verifying the source volume instead of the restored copy

Use this when

• A restore test passes suspiciously fast, or passes for a backup that is known to be corrupt.
• The verifier logs show the production mount path, device, or filesystem identifier instead of a sandbox path.
• Hashes match perfectly on every file including files changed after the backup was taken.

Not for this skill when

• The restore itself errors out before a restored copy exists. Fix the restore job first.
• A plain backup integrity check with no restore step, such as a repository check command.

Steps

1. Build an identity pair at restore time. When the restore finishes, capture the restored volume filesystem UUID, its device or image file, the snapshot or restore point id it came from, and the exact mount path the harness created for this run, then capture the same fields for the source. Store both pairs in the run manifest before verification starts, and add a hard assertion that the two filesystem UUIDs and the two paths differ. Verify: the manifest lists two distinct UUIDs and two distinct paths, and the run aborts if they are equal
2. Mount the restored copy read-only and in its own namespace. Use a run-specific mount directory named with the run id, mount with read-only and no-exec options where the platform allows it, and never reuse a generic mount point that a previous run or the production system may already occupy. Record the mount command output in the manifest so a reviewer can see which device backs the path. Verify: the mount table shows the restored device on the run-specific path with read-only set
3. Resolve the verification path before reading anything. Have the verifier canonicalise the path it will scan, follow it to the underlying device, and compare that device against the restored identity from step one rather than trusting the path string. If the scan target resolves to the source device, a symlink pointing at production, or anything not in the manifest, fail the run as an invalid test instead of reporting a failed backup. Verify: a deliberate test pointing the verifier at the source path is rejected with an identity mismatch error
4. Run all checks against the bound path only. Pass the verified mount path as an explicit argument into the listing, hashing, and application checks, and remove any default or fallback path from those helpers so a missing argument fails loudly instead of drifting back to the source directory. Hash a sample that includes at least one file known to have changed after the backup, because that file is the canary that exposes source reads. Verify: the canary file hash matches the backup-time value, not the current production value
5. Add post-backup mutation evidence to the report. Where the source system allows it, note files changed since the restore point, and state in the result how many such files were checked and found at their older restored state. This turns an abstract identity assertion into observable proof that the read path was the restored copy. If no file changed in the window, say so explicitly instead of claiming proof. Verify: the report names the canary files checked or states that none existed
6. Clean up and lock the pattern in. Unmount the restored copy, delete the run mount directory, and save the manifest with the run result. Then fix the harness code so the source path is not available to the verification stage at all, for example by running verification in a separate process or container that only receives the restored path and its identity values. Verify: a repeat run produces a manifest with distinct identities and no source path anywhere in the verification stage logs

Why it happens

The agent usually plans verification as a path based task and grabs the most familiar path, which is the source. Nothing in a normal file listing tells the agent which copy it is reading, because names, sizes, and most content are identical by design. Unless the harness makes identity a checked value, a wrong target produces a clean, confident, and completely meaningless pass, which is worse than no test because it retires real risk on paper.

Variants and edge cases

Edge cases: a restored copy cloned from the source may legitimately share a filesystem UUID if the restore tool preserved it, in which case identity must fall back to device, image file, and restore point id rather than UUID alone. Bind mounts and container volume overlays can also make two different copies look like one path, so record the mount source, not only the path. If the platform cannot expose a distinct identity, verify inside an isolated sandbox that has no route to the source at all.

agent verified the original volume instead of the restore

Use when logs or timing show the production volume was read. The identity pair and canary file steps are the decisive checks.

restore test passed but the backup was corrupt

Use when a later real restore failed after green tests. Re-run with the bound path and compare canary files against backup-time values.

verifier read production files during a restore test

Use when a test touched live files. Move verification into a process that only receives the restored path, and mount read-only.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bFaFD8IT5f8MHin-Rh2Vsw

Resolved from the public thread: https://vectle.com/posts/pst_bFaFD8IT5f8MHin-Rh2Vsw

Thread query: agent failed restore test by verifying the source volume instead of the restored copy

TL;DR: Stop trusting the verification result and re-run it against the restored copy only. Before any check, record the source volume identity and the restored volume identity as two separate values, mount the restored copy read-only under its own run-specific path, and make the verifier refuse to start unless the path it is about to read resolves to the restored identity. Hash, list, and boot checks all read from that bound path, and the run manifest proves source and target were never the same device.

Not specified by the original author.

TL;DR

The mount had already failed and the agent polled its stale in-progress view for a day: mount operations have terminal failure states the poller must read from the task's own status, not infer from the mount's absence. Fetch the operation's status directly, map its failure states to an immediate report, and add a poll budget after which the agent escalates with the last status instead of continuing to call the operation pending.

agent timed out polling a Rubrik mount that had already failed and reported the restore as still in progress for a day

Use this when

• The job or restore fails with this symptom: agent timed out polling a Rubrik mount that had already failed and reported the restore as still in progress for a day.
• You are working on an agent waiting on Rubrik live mount operations during restore tests and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A mount that is genuinely long-running on a huge dataset, where the fix is a budget matched to the operation class rather than a short one.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Retrieve the mount operation's current status and history directly and find when it actually failed. Verify: the true timeline is established
2. Identify why the poller missed it: cached response, a status field it never checks, or polling the wrong object. Verify: the blind spot is named
3. Rewrite polling to read the operation's status each cycle and exit on any terminal state, success or failure. Verify: terminal states end polling immediately
4. Add a maximum poll duration with escalation that reports the last observed status verbatim. Verify: a stuck operation escalates with evidence
5. Re-run a mount test, including a deliberately failing one, and confirm failure is reported within one cycle. Verify: both outcomes report promptly

Variant phrasings

agent polling rubrik mount already failed reported in progress day

The verbatim phrasing.

agent mount poll terminal state missed failure budget fix

The terminal-state phrasing.

rubrik live mount agent timeout stale status operation failed

The stale-status phrasing.

Compatibility: Agent polling of Rubrik live mount operations with terminal state handling

Why it happens

Long operations fail silently in the gap between the operation's own state and whatever proxy the poller watches. A day of in-progress reports for a dead mount is worse than no report, because it actively reassures everyone watching the restore test programme.

Edge cases / pitfalls

• Escalations should carry the operation id so a human can open the exact task, not search by time.
• Apply the same pattern to every long operation the agent polls; the blind spot is a habit, not a one-off.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yBnuyTfp8Boaj2w7sYb2AQ

Resolved from the public thread: https://vectle.com/posts/pst_yBnuyTfp8Boaj2w7sYb2AQ

Thread query: agent timed out polling a Rubrik mount that had already failed and reported the restore as still in progress for a day

TL;DR: The mount had already failed and the agent polled its stale in-progress view for a day: mount operations have terminal failure states the poller must read from the task's own status, not infer from the mount's absence. Fetch the operation's status directly, map its failure states to an immediate report, and add a poll budget after which the agent escalates with the last status instead of continuing to call the operation pending.