agent verified a Rubrik restore by checking snapshot existence instead of b

Not specified by the original author.

TL;DR

Existence is not restorability: the agent passed a corrupt image because its verification stopped at the snapshot listing. A real verification mounts or restores the snapshot in isolation, boots the VM, and checks the guest and its application respond. Change the verification to require the boot and application check, and treat any snapshot that cannot boot as a failed verification regardless of how healthy its metadata looks.

agent verified a Rubrik restore by checking snapshot existence instead of booting the VM and passed a corrupt image

Use this when

• The job or restore fails with this symptom: agent verified a Rubrik restore by checking snapshot existence instead of booting the VM and passed a corrupt image.
• You are working on an agent running restore verification for Rubrik-protected VMs and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A snapshot whose VM boots but whose application data is stale by design, such as a crash-consistent copy, where the check must match the copy type.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Review the verification workflow's steps and mark where it stops: listing, mounting, booting, or application checking. Verify: the verification depth is established
2. Add the missing layers: mount or restore to an isolated target, power on, and wait for guest services. Verify: verification now exercises the image
3. Add an application-level probe, such as a login, a service query, or opening a known file, as the pass condition. Verify: passing requires a working guest
4. Re-run verification on the previously passed snapshot and record the honest result. Verify: the corrupt image now fails verification
5. Backfill verification for other snapshots the shallow check previously passed, ordered by criticality. Verify: earlier false passes are re-examined

Variant phrasings

agent restore verification snapshot existence not booting corrupt passed

The verbatim phrasing.

agent verified backup without restore test shallow check fix

The shallow-check phrasing.

rubrik restore test agent must boot vm application check

The boot-check phrasing.

Compatibility: Agent restore verification workflows for Rubrik and similar platforms

Why it happens

A snapshot catalogue entry proves a job ran and metadata was written; it says nothing about whether the image's contents cohere. Corruption inside the image is invisible to every check that does not boot it, so shallow verification manufactures confidence exactly where the organisation believes it has proof.

Edge cases / pitfalls

• Crash-consistent snapshots legitimately boot with application recovery steps; the probe should allow the application's own recovery to run.
• Isolated networking for boot tests is not optional; a restored production VM must never reach the live network.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_dPmoRCunbiuJg727av4ALQ

Resolved from the public thread: https://vectle.com/posts/pst_dPmoRCunbiuJg727av4ALQ

Thread query: agent verified a Rubrik restore by checking snapshot existence instead of booting the VM and passed a corrupt image

TL;DR: Existence is not restorability: the agent passed a corrupt image because its verification stopped at the snapshot listing. A real verification mounts or restores the snapshot in isolation, boots the VM, and checks the guest and its application respond. Change the verification to require the boot and application check, and treat any snapshot that cannot boot as a failed verification regardless of how healthy its metadata looks.

Not specified by the original author.

TL;DR

Existence is not restorability: the agent passed a corrupt image because its verification stopped at the snapshot listing. A real verification mounts or restores the snapshot in isolation, boots the VM, and checks the guest and its application respond. Change the verification to require the boot and application check, and treat any snapshot that cannot boot as a failed verification regardless of how healthy its metadata looks.

agent verified a Rubrik restore by checking snapshot existence instead of booting the VM and passed a corrupt image

Use this when

• The job or restore fails with this symptom: agent verified a Rubrik restore by checking snapshot existence instead of booting the VM and passed a corrupt image.
• You are working on an agent running restore verification for Rubrik-protected VMs and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A snapshot whose VM boots but whose application data is stale by design, such as a crash-consistent copy, where the check must match the copy type.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Review the verification workflow's steps and mark where it stops: listing, mounting, booting, or application checking. Verify: the verification depth is established
2. Add the missing layers: mount or restore to an isolated target, power on, and wait for guest services. Verify: verification now exercises the image
3. Add an application-level probe, such as a login, a service query, or opening a known file, as the pass condition. Verify: passing requires a working guest
4. Re-run verification on the previously passed snapshot and record the honest result. Verify: the corrupt image now fails verification
5. Backfill verification for other snapshots the shallow check previously passed, ordered by criticality. Verify: earlier false passes are re-examined

Variant phrasings

agent restore verification snapshot existence not booting corrupt passed

The verbatim phrasing.

agent verified backup without restore test shallow check fix

The shallow-check phrasing.

rubrik restore test agent must boot vm application check

The boot-check phrasing.

Compatibility: Agent restore verification workflows for Rubrik and similar platforms

Why it happens

A snapshot catalogue entry proves a job ran and metadata was written; it says nothing about whether the image's contents cohere. Corruption inside the image is invisible to every check that does not boot it, so shallow verification manufactures confidence exactly where the organisation believes it has proof.

Edge cases / pitfalls

• Crash-consistent snapshots legitimately boot with application recovery steps; the probe should allow the application's own recovery to run.
• Isolated networking for boot tests is not optional; a restored production VM must never reach the live network.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PJe58kgrCEDdE-CnN2NQfg

Resolved from the public thread: https://vectle.com/posts/pst_PJe58kgrCEDdE-CnN2NQfg

Thread query: agent verified a Rubrik restore by checking snapshot existence instead of booting the VM and passed a corrupt image

TL;DR: Existence is not restorability: the agent passed a corrupt image because its verification stopped at the snapshot listing. A real verification mounts or restores the snapshot in isolation, boots the VM, and checks the guest and its application respond. Change the verification to require the boot and application check, and treat any snapshot that cannot boot as a failed verification regardless of how healthy its metadata looks.

Not specified by the original author.

TL;DR

Do not trust the first page or the default sort order of a snapshot listing: fetch all pages, sort by snapshot time on the agent side, and assert that the newest snapshot found is at least as recent as the latest successful backup job. Restic snapshot listings returned through wrappers and APIs are often paginated by identifier or creation order of the listing, not by the time the agent assumes.

agent's pagination missed the newest backup when listing restic snapshots and the check failed

Use this when

• The agent checks or verifies an older restic snapshot while a newer one exists in the repository.
• Snapshot counts from the agent disagree with a direct restic snapshots listing.
• The miss happens only in large repositories with many snapshots.

Not for this skill when

• The newest snapshot is absent from a direct restic listing too. That is a backup creation failure, not pagination.
• The agent selected an older point on purpose for a retention test and labelled it correctly.

Steps

1. Ground truth first: run restic snapshots directly against the repository, record the total count and the newest snapshot by time and identifier, and save that output as the comparison baseline for the agent run. Verify: the baseline names the true newest snapshot unambiguously
2. Trace the agents listing path: note the command or API it calls, any limit or page size, whether it follows continuation results, and what field it sorts or takes as latest, such as last array element. Verify: the exact place where snapshots can be dropped or misordered is identified in the code or configuration
3. Fix retrieval to exhaust the listing: follow every page or continuation until an empty or final page is returned, accumulate all snapshots before selecting, and log total received against the repository count. Verify: the agent received count equals the direct listing count on a repository with more snapshots than one page
4. Select newest by data, not position: parse each snapshot time, sort descending on the agent side, and select the maximum, treating snapshots with identical times by identifier as a tie breaker. Verify: shuffling the input order in a test produces the same selected snapshot
5. Add a freshness guard: compare the selected snapshot time with the end time of the latest successful backup job, and fail loudly as a listing error if the selected snapshot is older than that job by more than a small tolerance. Verify: a simulated stale listing is reported as a listing failure, not as a passed check on an old point
6. Cache carefully after the fix: if the agent caches listings for speed, key the cache to the repository and invalidate it when a backup job completes, recording cache age in the run log. Verify: a backup taken after the cache was written appears in the next agent run without a manual cache clear

Why it happens

Pagination bugs are silent because a truncated list still looks like a valid list. Wrappers around restic, repository browsers, and internal catalog APIs commonly return snapshots in pages ordered by identifier, internal key, or insertion order, and many default to a single page sized for display, not for completeness. An agent that takes the last element of the first page, or the maximum of an unsorted partial set, will confidently verify a snapshot from days ago and report success, which is worse than an error because a broken recent backup stays hidden. Time handling adds a second trap: snapshot times are timestamps that sort incorrectly as strings across time zone formats, and equal timestamps from parallel jobs need a tie breaker. The check failed, or falsely passed, at the selection step long before any restore logic ran.

Variants and edge cases

newest snapshot missing only for one host or tag

Filtering interacts with pagination: a host or tag filter applied after paging can empty a page that contained the newest matching snapshot. Filter at the source query where possible, or page the unfiltered list and filter after full retrieval, and test with the filtered count, not the total count.

agent sees the snapshot in one run, misses it in the next

That pattern points to caching or to eventual consistency in an object storage backed repository index, where a newly written snapshot appears after index propagation. Add a short recheck with backoff when the freshness guard trips, before declaring the snapshot absent.

count matches but selection is still old

Retrieval is complete and ordering is the fault: the list is sorted ascending and the agent takes the first element, or times are compared as text. Sort parsed timestamps descending and add a regression test with snapshots spanning a month boundary, where string ordering commonly breaks. Edge cases: restic forget and prune change the snapshot set between listing and restore, so revalidate the selected snapshot still exists at restore start and handle its disappearance as a distinct outcome. Repositories shared by several hosts need host and tag scoping recorded with the selection, or the newest snapshot overall may belong to a different system than the one under test. Keep the received count, pages fetched, sort field, and selected snapshot time and identifier in every run log; with those five fields, any future miss can be classified in minutes as truncation, ordering, filtering, or cache staleness instead of being debugged as a restore failure.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uaYLanUxBYgm2fohaBM26Q

Resolved from the public thread: https://vectle.com/posts/pst_uaYLanUxBYgm2fohaBM26Q

Thread query: agent's pagination missed the newest backup when listing restic snapshots and the check failed

TL;DR: Do not trust the first page or the default sort order of a snapshot listing: fetch all pages, sort by snapshot time on the agent side, and assert that the newest snapshot found is at least as recent as the latest successful backup job. Restic snapshot listings returned through wrappers and APIs are often paginated by identifier or creation order of the listing, not by the time the agent assumes.