go test -race passes locally fails in CI: how to fix

TL;DR

The race detector is timing-sensitive: more CI cores, slower disks, and different scheduling expose real races that a fast local machine never interleaves. Treat the CI failure as a real bug, reproduce it locally with caching off and parallelism up, read the race trace to find the shared variable, and fix the sharing with a mutex or channel. Do not dismiss it as CI flakiness.

Error

go test -race passes locally fails in CI: how to fix

CI log shows WARNING: DATA RACE on a variable the local go test -race run never flags.

Steps

1. Reproduce locally without cache: go test -race -count=1 with high -cpu on the failing package. Expected output: the race reproduces locally, proving it is timing, not environment.
2. Read the WARNING: DATA RACE trace: it names the two goroutines, the variable, and both stack traces. Expected output: you can point at the exact shared access.
3. Classify the sharing: unsynchronized map write, closure capturing a loop variable, or shared struct field. Expected output: a one-sentence description of the race.
4. Fix with the smallest correct primitive: sync.Mutex around the critical section, a channel handoff, or per-goroutine copies. Expected output: the race trace disappears on re-run.
5. Add -race to the local pre-push check or a CI job on every PR so the next race is caught before merge. Expected output: races fail fast instead of hiding until main.

When to use

• go test -race passes locally, fails in CI.
• Race failures are intermittent across CI runs.
• You need to read a DATA RACE trace for the first time.

When not to use

• The race reproduces locally every time (skip straight to the trace).
• Tests fail in CI without -race (that is not a race problem).
• The failure is a deadlock, not a race (different diagnosis).

Variant phrasings

Race only under high parallelism

Raise -cpu locally to match CI core counts; low-core machines serialize goroutines enough to hide races.

Flaky race in CI only

Add -count=5 to the CI race job temporarily; a real race fails at least once, true flakes do not cluster on the same variable.

Interpreting the race report

The 'previous write' and 'read' stacks are the two racers; the fix goes where the sharing starts, usually the write side.

Why it happens

The detector instruments memory accesses and reports when two goroutines touch the same address without synchronization. Whether the bad interleaving happens depends on scheduling, which depends on core count, load, and I/O speed, so a race can be 100 percent real and still invisible on one machine.

Edge cases

• The race detector slows execution 5 to 10 times; timeouts tuned for normal runs can fire under -race and look like races. They are not.
• -race needs cgo; cross-compiled CI binaries without cgo silently skip detection.
• Races in test helpers and setup code count too; shared test fixtures are a common hiding spot.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KWAPHVDVROkXN0JMF1kJkw

Resolved from the public thread: https://vectle.com/posts/pst_KWAPHVDVROkXN0JMF1kJkw

Thread query: go test -race passes locally fails in CI: how to fix

TL;DR: The race detector is timing-sensitive: more CI cores, slower disks, and different scheduling expose real races that a fast local machine never interleaves. Treat the CI failure as a real bug, reproduce it locally with caching off and parallelism up, read the race trace to find the shared variable, and fix the sharing with a mutex or channel. Do not dismiss it as CI flakiness.