agent's fix passed locally but introduced a new race in CI - it assumed single-threaded execution

TL;DR

The agent's fix assumed single-threaded execution, passed locally, and introduced a race that fails on parallel CI. Find the shared mutable state the fix touches, synchronize it (locks, atomic ops, or thread-local state), and verify under the same parallelism CI uses. Local serial runs cannot validate concurrent code.

The symptom

agent's fix passed locally but introduced a new race in CI  -  it assumed single-threaded execution

Steps

1. Reproduce under parallelism

Run the fixed test with CI's parallelism settings (workers, threads). The race should appear. Expected: Race reproduced under parallel execution.

2. Find the shared state

Look for what the fix reads and writes across threads: globals, caches, files, counters. Expected: The unsynchronized shared state identified.

3. Synchronize properly

Add locking, use atomic operations, or make the state thread-local. Match the granularity to the actual sharing. Expected: No data race under the race detector or stress run.

4. Verify at CI parallelism

Rerun with the full parallel configuration, ideally with a race detector enabled. Expected: Green under parallelism, not just serially.

When to use

• Fix passes locally, fails on parallel CI
• The fix touches shared mutable state
• Failures are intermittent and timing-shaped

When not to use

• CI runs serially too (different cause)
• The race predates the fix (fix it, but it is not the agent's regression)
• The code is provably thread-confined (verify, then look elsewhere)

Why it happens

Local runs are often serial or low-parallelism; CI runs wide. Code that is correct serially can be racy concurrently, and the agent validated in the easy environment. The race is in the fix, so the 'healed' test is now a new source of flakes.

Edge cases

• Test runners with process isolation: the race may be in shared files or DB rows, not memory.
• Flaky-by-design timing: replace sleeps with synchronization.
• Race detectors (tsan, -race): run them in CI for the touched packages.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AHJPKYzRo8PdFqV3rm7pHA