agent's checksum comparison compared compressed vs uncompressed hashes and failed every restore

TL;DR

The hashes disagree because they describe different byte streams, so define one canonical stream and hash only that. Decide whether the manifest hash covers the stored object exactly as written, or the original uncompressed content, label that choice in every manifest entry with the algorithm and the compression used, and make the verifier transform the restored side into the same domain before comparing. Add a known test vector for each backup job so a domain mix-up fails at setup time, not on every restore.

agent's checksum comparison compared compressed vs uncompressed hashes and failed every restore

Use this when

• Every restore from a job fails checksum comparison, yet the restored files open and behave correctly.
• Hashes match for uncompressed jobs and never match for compressed or deduplicated jobs.
• The two hash values change between backup and verify runs even when nothing was restored differently.

Not for this skill when

• Only some files in a restore fail, in a scattered pattern suggesting real damage.
• The backup repository check itself reports corrupt blocks.

Steps

1. Name the hash domain in the manifest. For each entry record the algorithm, the domain as either stored bytes or original content, the compression applied, and the expected length in that domain. Reject any comparison where the two sides carry different domain labels, and report it as a manifest mismatch rather than a corrupt file. Verify: a manifest entry shows algorithm, domain, compression, and length together
2. Canonicalise before comparing. When the domain is original content, the verifier must decompress the stored object with the recorded compression and hash the resulting stream, and must hash the restored file directly with no extra transformation. When the domain is stored bytes, compare the stored object hash before any decompression happens. Keep the transformation in one shared helper used by both backup and verify code so the two paths cannot drift apart. Verify: backup and verify call the same canonicalisation helper in the code path
3. Add a test vector per job. At job setup, back up one small known file, record its content hash computed by hand from the original bytes, and run a full restore and verify cycle on it. If the vector fails, the job configuration is wrong and no real restore point from that job should be trusted until it passes. Verify: the vector file passes end to end and its hand computed hash matches the manifest
4. Handle chunked and deduplicated storage explicitly. Where the store splits content into chunks, the canonical hash must be computed over the reassembled stream in order, not over chunk hashes concatenated in storage order if those differ. Record the chunk size and reassembly rule in the job notes, and test with a file larger than one chunk. Verify: a multi-chunk file verifies cleanly after a restore
5. Quarantine old manifests instead of failing their restores. Restore points written before the domain labelling existed should be verified with both domains attempted once, the matching domain recorded, and the entry relabelled for future runs. Report such points as verified under a legacy domain rather than as failures. Verify: a pre-fix restore point verifies after relabelling and shows its domain in the report
6. Fix the reporting language. Verification output must distinguish three outcomes, content match, content mismatch, and domain mismatch, because the operator response is completely different for each. Domain mismatches route to the harness fix in this skill and must not page as data loss. Verify: a deliberately mislabelled test entry reports domain mismatch and raises no corruption alert

Why it happens

A hash is only meaningful for the exact byte stream it was computed over, and compression changes that stream completely while preserving the content. Backup tools often hash what they wrote, which is compressed, while a naive agent verifier hashes what it restored, which is not. Every file then mismatches, including perfect restores, and the uniform failure pattern is the tell. Canonicalisation plus a stored domain label removes the ambiguity permanently instead of per incident.

Variants and edge cases

Edge cases: some tools compress per chunk with a dictionary, so decompressing a single chunk out of order gives wrong bytes and a false mismatch on large files only. Encrypted backups add a second transformation layer; the domain label must state whether hashing happens before encryption or after decryption. Files that were stored uncompressed because they were already compressed will pass under either naive check, which hides the bug until a compressible file appears.

restores fail checksum but files are fine

Use when restored content opens correctly despite hash failures. Compare domain labels before declaring corruption.

checksum mismatch on every file after enabling compression

Use when the failure started with a compression change. The backup side is hashing a different stream than the verifier.

hash values differ between backup and verify runs

Use when the same content yields changing hashes. Pin the algorithm, domain, and compression in the manifest and test with a vector file.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JPdkUsB4K7erJt-bv0m2Qw