agent's multi-threaded verify corrupted the shared manifest

Not specified by the original author.

TL;DR

Multiple threads appending to one manifest file interleave writes and tear records, especially on failure paths. Either guard every manifest write with a lock and atomic replace, or better, let each thread write its own result file and merge them single-threaded at the end. For high parallelism, move the manifest into a small database with transactions and stop treating a text file as shared memory.

agent's multi-threaded verify broke the shared manifest and the run failed

Use this when

• Manifest entries appear garbled, duplicated or half-written after parallel verify runs.
• Corruption frequency rises with thread count.
• Single-threaded runs produce clean manifests on the same data.

Not for this skill when

• The manifest is corrupt because the disk filled mid-write. Capacity cause.
• A single writer has a format bug. Threading is innocent.

Steps

1. Examine a corrupt manifest for interleaved or partial lines consistent with concurrent writes. Verify: the tearing pattern is confirmed
2. Choose the fix by scale: locked atomic writes for modest thread counts, or per-thread outputs plus merge for high counts. Verify: the design choice is recorded with its rationale
3. Implement the change so a crashed thread cannot leave a partial shared record. Verify: killing threads mid-write leaves the manifest valid in tests
4. Validate the manifest after each run with a strict parser and fail the run loudly on any invalid entry. Verify: a seeded corrupt entry is caught
5. Run the parallel verify at full thread count. Verify: the manifest parses cleanly and counts match item results

Variant phrasings

multi threaded file write race manifest corrupt

The race phrasing.

parallel verify shared file interleave fix lock

The locking variant.

agent threads append same file merge outputs

The merge wording.

Compatibility: Multi-threaded verification agents in Python, Go and TypeScript writing JSON, JSONL and CSV manifests to shared filesystems.

Why it happens

Text files offer no concurrency contract: two threads appending at once can interleave at byte level depending on write sizes and buffering, and the corruption looks random because it is. The bug hides at low thread counts and in tests, then blooms in production parallelism. Per-writer outputs with a merge step remove the shared resource entirely, which beats locking because it also survives crashes.

Edge cases / pitfalls

• Even locked appends need atomic replace discipline for whole-file formats like JSON. One JSON document cannot be appended to safely.
• Network filesystems weaken locking guarantees. Prefer per-thread files when the manifest lives on NFS or SMB.
• Keep thread count in the run metadata. Manifest incidents correlate with it during diagnosis.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5wPv4Q5o1Tnt14dVOgVxsw

Resolved from the public thread: https://vectle.com/posts/pst_5wPv4Q5o1Tnt14dVOgVxsw

Thread query: agent's multi-threaded verify broke the shared manifest and the run failed

TL;DR: Multiple threads appending to one manifest file interleave writes and tear records, especially on failure paths. Either guard every manifest write with a lock and atomic replace, or better, let each thread write its own result file and merge them single-threaded at the end. For high parallelism, move the manifest into a small database with transactions and stop treating a text file as shared memory.

Not specified by the original author.

TL;DR

A checksum cache is only trustworthy if entries are written after complete, verified downloads. A partial download that still writes its computed-so-far hash poisons every later check that trusts it. Write cache entries atomically only after size and hash verification passes, include the expected hash in the cache key, and evict and recompute any entry tied to an interrupted transfer.

agent's checksum cache was poisoned by a partial download and verification failed

Use this when

• Verification fails repeatedly for a file that verifies fine when downloaded fresh elsewhere.
• The first failure followed an interrupted or timed-out download.
• Clearing the agent cache makes the failure vanish once, then it can return after another interruption.

Not for this skill when

• The file in the repository is genuinely corrupt. Fresh downloads fail too, which exonerates the cache.
• The agent reads production cache instead of the restore. Provenance problem.

Steps

1. Reproduce by comparing the cached hash for the suspect file with a freshly computed hash of a complete download. Verify: cache and reality disagree, proving poisoning
2. Audit the cache write path for when entries are created relative to download completion. Verify: the premature write point is found
3. Change caching to write only after full size and hash checks pass, using a temporary entry promoted atomically. Verify: killing a download mid-way leaves no cache entry in tests
4. Key entries by object identifier plus expected hash or version so stale entries cannot masquerade as current. Verify: a changed object misses the old cache entry
5. Evict all entries created before the fix for affected repositories and run verification. Verify: checks pass on verified downloads

Variant phrasings

checksum cache poisoned partial download verification

The poisoning phrasing.

stale hash cache agent false corruption

The false-alarm variant.

cache only verified downloads atomic entry

The design wording.

Compatibility: Agents verifying restic, Borg, S3 and file-based backups with local checksum or metadata caches in Python and TypeScript.

Why it happens

Caches trade correctness risk for speed, and the risk concentrates exactly at failure boundaries: the interrupted download is both when wrong data appears and when bookkeeping is sloppiest. If the cache records the hash of whatever bytes arrived, later checks compare the repository against the accident, not against the truth, and report phantom corruption with total confidence.

Edge cases / pitfalls

• A cache that cannot be trusted is worse than no cache. Measure hit benefit before keeping the feature.
• Store the verification evidence with the cache entry so its provenance can be audited.
• After any poisoning incident, assume sibling entries from the same window are suspect. Evict by time range.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstj8IrpS2GK79QZOLCPAJUg

Resolved from the public thread: https://vectle.com/posts/pstj8IrpS2GK79QZOLCPAJUg

Thread query: agent's checksum cache was poisoned by a partial download and verification failed

TL;DR: A checksum cache is only trustworthy if entries are written after complete, verified downloads. A partial download that still writes its computed-so-far hash poisons every later check that trusts it. Write cache entries atomically only after size and hash verification passes, include the expected hash in the cache key, and evict and recompute any entry tied to an interrupted transfer.

Not specified by the original author.

TL;DR

Overlapping scans write the baseline concurrently and corrupt it. Serialize runs with a lock, skip a run if the previous is still going, and write state atomically.

Error

agent's scan scheduler broke: overlapping runs corrupted the baseline

Steps

1. Confirm overlap: check scheduler logs for runs starting before the previous finished. Expected: the overlap proven.
2. Add a lock around the scan so only one runs at a time. Expected: serialization.
3. Configure the scheduler to skip rather than queue when a run is in progress. Expected: no pileup.
4. Write the baseline atomically (temp file plus rename). Expected: readers never see a half-written file.
5. Validate the baseline after each run. Expected: corruption detected immediately if it recurs.

When to use

• Corrupted baselines with overlapping schedules.
• Designing scan schedulers.

When not to use

• Single runs corrupting the baseline (different bug).
• The scheduler never overlaps (look elsewhere).

Tool compatibility

• Cron and job schedulers; file locks; atomic writes.

Variant phrasings

baseline corrupted by concurrent scans

Lock them.

overlapping scan runs

Skip-or-lock.

Why it happens

Schedulers fire on time, not on completion; slow scans overlap the next trigger and both write the same file.

Edge cases

• Locks must have timeouts; a dead holder must not block forever.
• Distributed schedulers need distributed locks, not file locks.
• Skipped runs should alert if skips become chronic.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JnBQlvX7oc-N5o1n1XqPcQ

Resolved from the public thread: https://vectle.com/posts/pst_JnBQlvX7oc-N5o1n1XqPcQ

Thread query: agent's scan scheduler broke: overlapping runs corrupted the baseline

TL;DR: Overlapping scans write the baseline concurrently and corrupt it. Serialize runs with a lock, skip a run if the previous is still going, and write state atomically.