go test fatal error: concurrent map writes flake in CI - how to debug

TL;DR

Two goroutines wrote to the same Go map at the same time, which is a fatal runtime error, not a recoverable panic. It flakes in CI because higher parallelism changes goroutine interleaving. Run the suite with -race: the race detector prints both goroutine stacks with file and line numbers. Then serialize the access with a mutex, switch to sync.Map, or give one goroutine ownership of the map.

The query

go test fatal error: concurrent map writes flake in CI  -  how to debug

Use this when

• go test dies intermittently with fatal error: concurrent map writes
• the failure moves between runs and never reproduces single-threaded
• the stack mentions runtime.mapassign or concurrent map

Not for

• a data race on a slice, struct field, or counter (use -race the same way, but the fix differs)
• a deadlock where nothing progresses (no fatal error is printed; use the timeout dump instead)
• the race is in test helper code rather than production code (still fix it; flaky helpers poison the suite)

Steps

1. Rerun with the race detector

The race detector turns the fatal error into a detailed report naming both parties. Run the suspect package with it.

go test -race -count=1 ./internal/cache/

Expected output: the output contains WARNING: DATA RACE with two goroutine stacks.

2. Read both stacks to find the shared map

The report shows the write that raced and the other goroutine's access. The map variable and both call sites are named.

WARNING: DATA RACE
Write at 0x... by goroutine 12:
  internal/cache.(*Store).Set(...)
      /app/internal/cache/store.go:44
Previous write at 0x... by goroutine 9:
  internal/cache.(*Store).Set(...)
      /app/internal/cache/store.go:44

Expected output: both racing call sites are identified with file and line.

3. Serialize the access

Pick the cheapest correct fix: a mutex around the critical section for general use, sync.Map for mostly-read workloads, or channel-based ownership.

type Store struct {
    mu sync.RWMutex
    m  map[string]Item
}
func (s *Store) Set(k string, v Item) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.m[k] = v
}

Expected output: the code compiles and the critical section is covered.

4. Verify with -race and add it to CI

Run the full package with the detector and make it part of the CI job so the next race is caught at introduction time.

go test -race -count=1 ./...

Expected output: -race is clean across the module and the CI job runs the detector on every push.

Tool compatibility

• Go 1.21 - 1.24
• race detector requires cgo (a C toolchain in the CI image)

Variant phrasings

• fatal error: concurrent map writes flake in CI
• go test WARNING: DATA RACE on map
• concurrent map iteration and map write in CI

Why it happens

Go maps are not safe for concurrent use by design: the runtime deliberately crashes instead of corrupting memory silently. Local runs often pass because fewer CPUs and less parallelism produce a lucky interleaving every time. CI runners with more cores and parallel packages reshuffle the interleaving until the race fires.

Edge cases / pitfalls

• the race detector needs gcc in the CI image; a missing toolchain fails the build, not the test
• sync.Map is only faster for append-mostly or read-mostly patterns with stable keys; a mutex wins for general use
• a race that only appears with -count=2 or higher is still real; loop the suite a few times before declaring victory
• maps guarded by a mutex must never escape the guard: returning the map itself reintroduces the race

Provenance

Resolved from the public thread: https://vectle.com/posts/pst92qiuxBKY-W_fzTuGLKwA