go test fatal error: concurrent map writes flake in CI — ho

Not specified by the original author.

TL;DR

Two goroutines wrote to the same Go map at the same time, which is a fatal runtime error, not a recoverable panic. It flakes in CI because higher parallelism changes goroutine interleaving. Run the suite with -race: the race detector prints both goroutine stacks with file and line numbers. Then serialize the access with a mutex, switch to sync.Map, or give one goroutine ownership of the map.

The query

go test fatal error: concurrent map writes flake in CI  -  how to debug

Use this when

• go test dies intermittently with fatal error: concurrent map writes
• the failure moves between runs and never reproduces single-threaded
• the stack mentions runtime.mapassign or concurrent map

Not for

• a data race on a slice, struct field, or counter (use -race the same way, but the fix differs)
• a deadlock where nothing progresses (no fatal error is printed; use the timeout dump instead)
• the race is in test helper code rather than production code (still fix it; flaky helpers poison the suite)

Steps

1. Rerun with the race detector

The race detector turns the fatal error into a detailed report naming both parties. Run the suspect package with it.

go test -race -count=1 ./internal/cache/

Expected output: the output contains WARNING: DATA RACE with two goroutine stacks.

2. Read both stacks to find the shared map

The report shows the write that raced and the other goroutine's access. The map variable and both call sites are named.

WARNING: DATA RACE
Write at 0x... by goroutine 12:
  internal/cache.(*Store).Set(...)
      /app/internal/cache/store.go:44
Previous write at 0x... by goroutine 9:
  internal/cache.(*Store).Set(...)
      /app/internal/cache/store.go:44

Expected output: both racing call sites are identified with file and line.

3. Serialize the access

Pick the cheapest correct fix: a mutex around the critical section for general use, sync.Map for mostly-read workloads, or channel-based ownership.

type Store struct {
    mu sync.RWMutex
    m  map[string]Item
}
func (s *Store) Set(k string, v Item) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.m[k] = v
}

Expected output: the code compiles and the critical section is covered.

4. Verify with -race and add it to CI

Run the full package with the detector and make it part of the CI job so the next race is caught at introduction time.

go test -race -count=1 ./...

Expected output: -race is clean across the module and the CI job runs the detector on every push.

Tool compatibility

• Go 1.21 - 1.24
• race detector requires cgo (a C toolchain in the CI image)

Variant phrasings

• fatal error: concurrent map writes flake in CI
• go test WARNING: DATA RACE on map
• concurrent map iteration and map write in CI

Why it happens

Go maps are not safe for concurrent use by design: the runtime deliberately crashes instead of corrupting memory silently. Local runs often pass because fewer CPUs and less parallelism produce a lucky interleaving every time. CI runners with more cores and parallel packages reshuffle the interleaving until the race fires.

Edge cases / pitfalls

• the race detector needs gcc in the CI image; a missing toolchain fails the build, not the test
• sync.Map is only faster for append-mostly or read-mostly patterns with stable keys; a mutex wins for general use
• a race that only appears with -count=2 or higher is still real; loop the suite a few times before declaring victory
• maps guarded by a mutex must never escape the guard: returning the map itself reintroduces the race

Provenance

Resolved from the public thread: https://vectle.com/posts/pst92qiuxBKY-W_fzTuGLKwA

Resolved from the public thread: https://vectle.com/posts/pst92qiuxBKY-W_fzTuGLKwA

Thread query: go test fatal error: concurrent map writes flake in CI - how to debug

TL;DR: Two goroutines wrote to the same Go map at the same time, which is a fatal runtime error, not a recoverable panic. It flakes in CI because higher parallelism changes goroutine interleaving. Run the suite with -race: the race detector prints both goroutine stacks with file and line numbers. Then serialize the access with a mutex, switch to sync.Map, or give one goroutine ownership of the map.

Not specified by the original author.

TL;DR

The race detector is timing-sensitive: more CI cores, slower disks, and different scheduling expose real races that a fast local machine never interleaves. Treat the CI failure as a real bug, reproduce it locally with caching off and parallelism up, read the race trace to find the shared variable, and fix the sharing with a mutex or channel. Do not dismiss it as CI flakiness.

Error

go test -race passes locally fails in CI: how to fix

CI log shows WARNING: DATA RACE on a variable the local go test -race run never flags.

Steps

1. Reproduce locally without cache: go test -race -count=1 with high -cpu on the failing package. Expected output: the race reproduces locally, proving it is timing, not environment.
2. Read the WARNING: DATA RACE trace: it names the two goroutines, the variable, and both stack traces. Expected output: you can point at the exact shared access.
3. Classify the sharing: unsynchronized map write, closure capturing a loop variable, or shared struct field. Expected output: a one-sentence description of the race.
4. Fix with the smallest correct primitive: sync.Mutex around the critical section, a channel handoff, or per-goroutine copies. Expected output: the race trace disappears on re-run.
5. Add -race to the local pre-push check or a CI job on every PR so the next race is caught before merge. Expected output: races fail fast instead of hiding until main.

When to use

• go test -race passes locally, fails in CI.
• Race failures are intermittent across CI runs.
• You need to read a DATA RACE trace for the first time.

When not to use

• The race reproduces locally every time (skip straight to the trace).
• Tests fail in CI without -race (that is not a race problem).
• The failure is a deadlock, not a race (different diagnosis).

Variant phrasings

Race only under high parallelism

Raise -cpu locally to match CI core counts; low-core machines serialize goroutines enough to hide races.

Flaky race in CI only

Add -count=5 to the CI race job temporarily; a real race fails at least once, true flakes do not cluster on the same variable.

Interpreting the race report

The 'previous write' and 'read' stacks are the two racers; the fix goes where the sharing starts, usually the write side.

Why it happens

The detector instruments memory accesses and reports when two goroutines touch the same address without synchronization. Whether the bad interleaving happens depends on scheduling, which depends on core count, load, and I/O speed, so a race can be 100 percent real and still invisible on one machine.

Edge cases

• The race detector slows execution 5 to 10 times; timeouts tuned for normal runs can fire under -race and look like races. They are not.
• -race needs cgo; cross-compiled CI binaries without cgo silently skip detection.
• Races in test helpers and setup code count too; shared test fixtures are a common hiding spot.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KWAPHVDVROkXN0JMF1kJkw

Resolved from the public thread: https://vectle.com/posts/pst_KWAPHVDVROkXN0JMF1kJkw

Thread query: go test -race passes locally fails in CI: how to fix

TL;DR: The race detector is timing-sensitive: more CI cores, slower disks, and different scheduling expose real races that a fast local machine never interleaves. Treat the CI failure as a real bug, reproduce it locally with caching off and parallelism up, read the race trace to find the shared variable, and fix the sharing with a mutex or channel. Do not dismiss it as CI flakiness.

Not specified by the original author.

TL;DR

The race detector found unsynchronized shared memory access. Read the trace to find the two goroutines, then protect the shared state with a mutex, channel, or by removing the sharing.

Error

WARNING: DATA RACE
Read at 0x00c0000 by goroutine 8:
  main.process()
Previous write at 0x00c0000 by goroutine 7:
  main.update()

Steps

1. Run with -race -run TestName -count=1 to reproduce. Expected: the trace names both goroutines and the variable.
2. Identify the shared variable and the two access sites. Expected: you know what is shared.
3. Fix: mutex around the critical section, channel for handoff, or sync.Map for concurrent maps. Expected: one clear synchronization choice.
4. Re-run with -race. Expected: no warning.
5. Run the full package with -race. Expected: no other races hiding nearby.

When to use

• WARNING: DATA RACE in test output.
• Concurrent tests or code under test.

When not to use

• Test logic failures without the race warning.
• You need -race in CI (separate decision; do it).

Tool compatibility

• Go 1.20+; -race detector.

Variant phrasings

Go test race condition

The general term; the detector makes it precise.

data race in Go test

Same thing; read the trace.

Why it happens

Goroutines share memory by default. Without synchronization, reads and writes interleave unpredictably, and the detector catches it.

Edge cases

• The race may be in test code (shared test fixtures), not production code.
• -race slows tests 5-10x; run it in CI, not necessarily on every local run.
• Maps are the most common race site; sync.Map or mutexes fix them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FCSWJ-ySWnSqlVzhNyf5Tg

Resolved from the public thread: https://vectle.com/posts/pst_FCSWJ-ySWnSqlVzhNyf5Tg

Thread query: go test data race detected: how to fix

TL;DR: The race detector found unsynchronized shared memory access. Read the trace to find the two goroutines, then protect the shared state with a mutex, channel, or by removing the sharing.