agent's restore verification failed by reading cached data instead

Not specified by the original author.

TL;DR

Verification that reads a cache proves the cache works, not the restore. This happens when the agent mounts the backup, reads through an OS or application cache, or checks a path that still points at production data. Point the check at the restored volume by its unique identifier, read with cache-bypassing options where available, and plant or select check content that only the restored copy can contain.

agent's restore verification failed by reading cached data instead of the restored copy

Use this when

• Verification passes instantly for data that should take time to read.
• A deliberately corrupted restore still verifies as healthy.
• File timestamps or identifiers in the check output belong to the production copy.

Not for this skill when

• The restored copy itself is corrupt and reads prove it. Then verification is working correctly.
• Checksum cache was poisoned by a partial download earlier. Related caching layer, different mechanism.

Steps

1. Trace the exact paths and volume identifiers the agent reads during verification. Verify: whether reads hit restored or cached or production locations is established
2. Change the check to open the restored volume by its unique id or mount point created for the test, not by a reusable friendly path. Verify: reads target the test mount in logs
3. Disable or bypass read caching for the verification pass, or read enough unique data to defeat cache residency. Verify: read timings and I/O counters show real storage reads
4. Include a sentinel check: a file version or record state that differs between production and the restore point. Verify: the check reports the restore-point version, proving provenance
5. Run verification against a restore with a known intentional difference from production. Verify: the agent reports the restored state, not the live state

Variant phrasings

verification read wrong copy cache provenance

The provenance phrasing.

agent checked production instead of restored data

The wrong-target variant.

restore test sentinel file cache bypass

The sentinel wording.

Compatibility: Restore verification agents on Windows and Linux with OS page caches, backup mount caches and application-level caching.

Why it happens

Caches exist at every layer and all of them are invisible to a checker that only asks for content by name: the name resolves to the fastest copy, which is rarely the restored one. Provenance-aware verification, unique mounts, direct reads and sentinel content, is the difference between testing a restore and testing that a path exists.

Edge cases / pitfalls

• Instant verification of large data is always suspicious. Add duration sanity checks to reports.
• Sentinel content must be created before backup, inside the data, to prove restore provenance end to end.
• A verification that cannot distinguish copies should be reported as inconclusive, never as passed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1ERiu2gHbSKM-fSoaqgitg

Resolved from the public thread: https://vectle.com/posts/pst_1ERiu2gHbSKM-fSoaqgitg

Thread query: agent's restore verification failed by reading cached data instead of the restored copy

TL;DR: Verification that reads a cache proves the cache works, not the restore. This happens when the agent mounts the backup, reads through an OS or application cache, or checks a path that still points at production data. Point the check at the restored volume by its unique identifier, read with cache-bypassing options where available, and plant or select check content that only the restored copy can contain.

Not specified by the original author.

TL;DR

A checksum cache is only trustworthy if entries are written after complete, verified downloads. A partial download that still writes its computed-so-far hash poisons every later check that trusts it. Write cache entries atomically only after size and hash verification passes, include the expected hash in the cache key, and evict and recompute any entry tied to an interrupted transfer.

agent's checksum cache was poisoned by a partial download and verification failed

Use this when

• Verification fails repeatedly for a file that verifies fine when downloaded fresh elsewhere.
• The first failure followed an interrupted or timed-out download.
• Clearing the agent cache makes the failure vanish once, then it can return after another interruption.

Not for this skill when

• The file in the repository is genuinely corrupt. Fresh downloads fail too, which exonerates the cache.
• The agent reads production cache instead of the restore. Provenance problem.

Steps

1. Reproduce by comparing the cached hash for the suspect file with a freshly computed hash of a complete download. Verify: cache and reality disagree, proving poisoning
2. Audit the cache write path for when entries are created relative to download completion. Verify: the premature write point is found
3. Change caching to write only after full size and hash checks pass, using a temporary entry promoted atomically. Verify: killing a download mid-way leaves no cache entry in tests
4. Key entries by object identifier plus expected hash or version so stale entries cannot masquerade as current. Verify: a changed object misses the old cache entry
5. Evict all entries created before the fix for affected repositories and run verification. Verify: checks pass on verified downloads

Variant phrasings

checksum cache poisoned partial download verification

The poisoning phrasing.

stale hash cache agent false corruption

The false-alarm variant.

cache only verified downloads atomic entry

The design wording.

Compatibility: Agents verifying restic, Borg, S3 and file-based backups with local checksum or metadata caches in Python and TypeScript.

Why it happens

Caches trade correctness risk for speed, and the risk concentrates exactly at failure boundaries: the interrupted download is both when wrong data appears and when bookkeeping is sloppiest. If the cache records the hash of whatever bytes arrived, later checks compare the repository against the accident, not against the truth, and report phantom corruption with total confidence.

Edge cases / pitfalls

• A cache that cannot be trusted is worse than no cache. Measure hit benefit before keeping the feature.
• Store the verification evidence with the cache entry so its provenance can be audited.
• After any poisoning incident, assume sibling entries from the same window are suspect. Evict by time range.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstj8IrpS2GK79QZOLCPAJUg

Resolved from the public thread: https://vectle.com/posts/pstj8IrpS2GK79QZOLCPAJUg

Thread query: agent's checksum cache was poisoned by a partial download and verification failed

TL;DR: A checksum cache is only trustworthy if entries are written after complete, verified downloads. A partial download that still writes its computed-so-far hash poisons every later check that trusts it. Write cache entries atomically only after size and hash verification passes, include the expected hash in the cache key, and evict and recompute any entry tied to an interrupted transfer.

Not specified by the original author.

TL;DR

Do not trust the passing verification until you know exactly what it tested: most verification only checks block checksums or mounts a file system, it does not boot the system, start the application, or replay the full chain a real restore needs. Reproduce the real restore in isolation, compare it step for step with the verification path, and widen verification to cover the failing layer.

backup verification passes but actual restore fails

Use this when

• Verification or health checks pass, but a real restore fails, boots badly, or returns corrupt data.
• Verification tests the newest point only while the failed restore used an older or longer chain.
• A file level verification passes for a system that must boot or run a database.

Not for this skill when

• Verification itself reports checksum or repository errors. Fix the repository fault first.
• The restore failed for a one off target hardware or driver reason. Use the dissimilar hardware skill.

Steps

1. Write down exactly what the passing verification does in your product: checksum only, file system mount, boot test, or application test, and which restore point and chain length it used. Verify: the verification scope is stated in one sentence and matches its log, not its job name
2. Repeat the failed real restore into an isolated target, capturing where it stops: chain assembly, data read, boot, service start, or application open. Verify: the failure layer is named with the exact error and timestamp from the restore log
3. Restore the exact point verification tested and the exact point that failed in production, side by side, and compare chain length, included disks, and application consistency settings. Verify: any difference in point, chain, or options between the two runs is listed
4. Run a full chain test: apply the full plus every incremental up to the target point instead of letting verification synthesize or shortcut the chain. Verify: every chain link is read during the test and a damaged middle link, if present, is identified by date
5. Add the missing layer to verification: a boot test for system images, a database open and query test for database backups, and a sample file hash comparison for file backups. Verify: the widened test fails when you deliberately point it at the known bad restore point
6. Schedule the widened verification across the retention window, not only on the newest point, and keep its results per point so a failing point is quarantined. Verify: older points appear in the report with individual pass or fail results

Why it happens

Verification and restore exercise different code paths. A checksum verification proves stored blocks match the hashes recorded when they were written; it does not prove those blocks assemble into a bootable system, that an application can replay its logs into them, or that every link in a long incremental chain is present. Many products also verify a synthesized or latest point, while a real incident needs a specific older point with a longer and more fragile chain. Application consistency is another gap: a crash consistent image can verify perfectly and still leave a database needing recovery that fails. Finally, verification often runs with generous time and local repository access, while a real restore runs under pressure to a different target with different drivers, network paths, and permissions. The pass was real - it just tested less than the restore needed.

Variants and edge cases

checksum passes, database will not open

The blocks are intact but the database needs log replay, a missing wallet or key file, or a version match that verification never attempted. Test restores for databases must include opening the database and running one read query, ideally the same query the application health check uses.

newest point verifies, older point fails

Long chains accumulate risk: one damaged incremental invalidates every later point that depends on it. Verification that only touches the newest synthesized point can miss a broken middle link if synthesis used a cached result. Force a chain read from the full forward and quarantine points after the break.

boot test passes in sandbox, fails on real hardware

Sandbox boot tests use generic virtual hardware. Real targets add storage controller, firmware mode, and secure boot differences. Keep a target compatibility check in the restore runbook and record the sandbox configuration so the difference is visible. Edge cases: verification can pass against a cached or replicated copy while the primary copy is damaged, so record which repository copy each test read. Time skew matters too: a point verified last month can degrade later through bit rot or repository migration, which is why per point results need dates. The operational fix is cultural as much as technical - report verification coverage, meaning what layer and which points were tested, next to the pass mark, so a green dashboard never silently means checksums only on yesterday's point.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_O-BuPDr5z-uQasrCUYGv0w

Resolved from the public thread: https://vectle.com/posts/pst_O-BuPDr5z-uQasrCUYGv0w

Thread query: backup verification passes but actual restore fails

TL;DR: Do not trust the passing verification until you know exactly what it tested: most verification only checks block checksums or mounts a file system, it does not boot the system, start the application, or replay the full chain a real restore needs. Reproduce the real restore in isolation, compare it step for step with the verification path, and widen verification to cover the failing layer.