agent's multi-threaded verify broke the shared manifest and the run failed

TL;DR

Multiple threads appending to one manifest file interleave writes and tear records, especially on failure paths. Either guard every manifest write with a lock and atomic replace, or better, let each thread write its own result file and merge them single-threaded at the end. For high parallelism, move the manifest into a small database with transactions and stop treating a text file as shared memory.

agent's multi-threaded verify broke the shared manifest and the run failed

Use this when

• Manifest entries appear garbled, duplicated or half-written after parallel verify runs.
• Corruption frequency rises with thread count.
• Single-threaded runs produce clean manifests on the same data.

Not for this skill when

• The manifest is corrupt because the disk filled mid-write. Capacity cause.
• A single writer has a format bug. Threading is innocent.

Steps

1. Examine a corrupt manifest for interleaved or partial lines consistent with concurrent writes. Verify: the tearing pattern is confirmed
2. Choose the fix by scale: locked atomic writes for modest thread counts, or per-thread outputs plus merge for high counts. Verify: the design choice is recorded with its rationale
3. Implement the change so a crashed thread cannot leave a partial shared record. Verify: killing threads mid-write leaves the manifest valid in tests
4. Validate the manifest after each run with a strict parser and fail the run loudly on any invalid entry. Verify: a seeded corrupt entry is caught
5. Run the parallel verify at full thread count. Verify: the manifest parses cleanly and counts match item results

Variant phrasings

multi threaded file write race manifest corrupt

The race phrasing.

parallel verify shared file interleave fix lock

The locking variant.

agent threads append same file merge outputs

The merge wording.

Compatibility: Multi-threaded verification agents in Python, Go and TypeScript writing JSON, JSONL and CSV manifests to shared filesystems.

Why it happens

Text files offer no concurrency contract: two threads appending at once can interleave at byte level depending on write sizes and buffering, and the corruption looks random because it is. The bug hides at low thread counts and in tests, then blooms in production parallelism. Per-writer outputs with a merge step remove the shared resource entirely, which beats locking because it also survives crashes.

Edge cases / pitfalls

• Even locked appends need atomic replace discipline for whole-file formats like JSON. One JSON document cannot be appended to safely.
• Network filesystems weaken locking guarantees. Prefer per-thread files when the manifest lives on NFS or SMB.
• Keep thread count in the run metadata. Manifest incidents correlate with it during diagnosis.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5wPv4Q5o1Tnt14dVOgVxsw