the agent's bisection found no culprit because the order-dependency needs three specific tests in sequence

TL;DR Single-test bisection fails when the bug needs three tests in sequence, because removing any one of them hides it. Bisect over groups and sequences instead of individual tests.

the agent's bisection found no culprit because the order-dependency needs three specific tests in sequence

Steps

1. Confirm the multi-test shape. Verify that removing ANY single test from the failing selection makes the failure disappear; that is the signature of a sequence dependency. Expected: no individual test is sufficient to reproduce the failure.

2. Find the minimal failing sequence. Start from the full failing selection and remove tests from the front and back in chunks; keep whatever removal still fails until you cannot remove any more. Expected: a minimal ordered list, often exactly three tests, that still reproduces the failure.

3. Understand the chain. For the minimal sequence, identify what each test contributes: test A sets up state, test B transforms it, test C observes the corruption. Write the chain down. Expected: a three-link causal story, e.g. "A creates the row, B moves it to a bad state, C asserts on it".

4. Break the chain at the cheapest link. Usually that means proper teardown in A or B so the state never reaches C corrupted; sometimes C needs isolation from the shared world. Expected: the minimal sequence passes, and the full suite passes under the original failing order.

5. Record the sequence as a regression test or a documented ordering case. Expected: the suite now pins this interaction so it cannot silently return.

Use this when

• bisection over single tests found nothing
• the failure needs a specific multi-test sequence to trigger
• removing any one test from the selection hides the failure
• the agent concluded "no culprit" and stopped

Not for this skill when

• a single polluter explains the failure (normal bisect works)
• the failure is timing-based rather than order-based
• the sequence is intentional fixture chaining (by design)
• the failure reproduces with a single test standalone

Variant phrasings

• "bisection found no culprit for order dependent failure"
• "test failure needs three tests in sequence to reproduce"
• "how to bisect multi-test ordering dependencies"

Why it happens

Classic bisection assumes one culprit: remove half, see if it still fails, repeat. With a three-test chain, every halving removes at least one link, so the failure vanishes at every step and the bisect reports "no culprit". The agent trusts the tool's answer instead of questioning the tool's assumption.

Edge cases

• The minimal sequence may be longer than three; the chunk-removal method still converges, just slower.
• xdist can split the sequence across workers and hide it; reproduce serially first, then confirm under the CI worker count.
• If the chain involves randomness (a seed-dependent middle step), fix the seed before bisecting or the minimal sequence will not be stable.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstG0C2r5vwyMnJYgEco-x7A