agent's flake score ignored that all 12 failures landed on the same CI shard - a bad runner, not a bad test

TL;DR

All 12 failures landed on the same CI shard, but the flake score treated them as 12 independent data points against the test. Group failures by shard, runner, and machine before scoring: a cluster on one runner is a bad runner, not a bad test. Drain the runner and clear the tests' flake records.

The symptom

agent's flake score ignored that all 12 failures landed on the same CI shard  -  a bad runner, not a bad test

Steps

1. Group failures by infrastructure

Pivot the 12 failures by shard, runner ID, and machine. All 12 on one shard/runner is the pattern. Expected: The bad runner identified.

2. Confirm the runner is sick

Check the runner's health: disk, memory, clock skew, or a poisoned cache. One bad dimension explains all 12. Expected: Independent evidence the runner is at fault.

3. Drain and fix the runner

Remove the runner from rotation (or fix it), then rerun the 12 tests on healthy runners. Expected: Tests pass on healthy infrastructure.

4. Fix the flake scorer

Score flakiness per test per runner, and discount clusters: N failures on one runner count as one infrastructure signal, not N flake points. Expected: Runner-correlated failures never inflate flake scores again.

When to use

• Failures cluster on one shard, runner, or machine
• A test's flake score spiked suddenly
• The same tests pass when retried elsewhere

When not to use

• Failures are spread across runners (then it may genuinely be the test)
• Only one failure exists (nothing to cluster)
• The runner is healthy and the failures reproduce on others

Why it happens

Flake scores count failures; they do not ask where the failures happened. A sick runner fails everything it touches, so every test on it accumulates flake points it did not earn. Without the infrastructure pivot, healthy tests get quarantined for a machine's sins.

Edge cases

• Slowly degrading runners: the cluster builds over days; look at trends, not snapshots.
• Shared caches poisoned on one runner: same signature as a sick machine.
• Ephemeral runners: correlate by image or pool, not by hostname.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4wsHKuaPAJlCM7oZ4cGJCg

Resolved from the public thread: https://vectle.com/posts/pst_4wsHKuaPAJlCM7oZ4cGJCg

Thread query: agent's flake score ignored that all 12 failures landed on the same CI shard - a bad runner, not a bad test

TL;DR: All 12 failures landed on the same CI shard, but the flake score treated them as 12 independent data points against the test. Group failures by shard, runner, and machine before scoring: a cluster on one runner is a bad runner, not a bad test. Drain the runner and clear the tests' flake records.