agent bisected with a different random seed than CI uses - the order-dependency never appeared in its repro

TL;DR

The agent bisected with a different random seed than CI, so the order-dependency never appeared. Capture the exact seed and ordering from the failing CI run, reproduce with those identical settings, then bisect. Bisection is only valid under the same seed that produced the failure.

The symptom

agent bisected with a different random seed than CI uses  -  the order-dependency never appeared in its repro

Steps

1. Capture CI's seed and ordering

Read the failing CI log for the random seed (pytest --randomly-seed, rspec --seed, jest --seed) and the test order. If the runner does not log the seed, add seed logging first. Expected: The exact seed value from the failing run.

2. Reproduce with the identical seed

Run the suite locally with the same seed and the same ordering options. The order-dependent failure should appear. Expected: The CI failure reproduced under the captured seed.

3. Bisect under the fixed seed

Keep the seed pinned while bisecting: remove half the tests, rerun with the same seed, and narrow down the interacting pair. Changing the seed mid-bisect invalidates the search. Expected: The minimal test pair that triggers the order dependency.

4. Make seeds first-class in the healer

Teach the agent to always capture and reuse the failing run's seed before attempting any reproduction or bisection. Expected: No more seed-mismatched repro attempts.

When to use

• An order-dependent failure will not reproduce for the agent
• CI uses randomized ordering with a seed
• Bisection keeps coming back clean

When not to use

• The runner does not randomize order (then the failure should reproduce directly)
• The failure is seed-independent (different cause)
• The seed is already matched and it still will not reproduce (check environment next)

Why it happens

Randomized ordering exists to surface order dependencies, but each seed produces a different order. Bisecting under a different seed explores a different order space, so the specific interacting pair never meets. The seed is part of the reproduction, not a detail.

Edge cases

• Seeds that only affect a subset (e.g. per-file shuffling): match the granularity too.
• Non-deterministic tests that fail under every seed: the seed is not the issue; the test is.
• CI systems that do not expose the seed: pin one explicitly in the CI config.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AzC33iqu9dbj8YtBqt7g5Q