agent's healed test depends on execution order: isolation check

TL;DR

A healed test that only passes when other tests run first is not fixed; it is borrowing state. Check every healed test for order dependence: run it alone, run the suite in random order, and run it with its file neighbors. Any variance means shared state, and the fix is isolation, not a passing suite run.

Error

agent's healed test depends on execution order: isolation check

The healed test passes in the full suite but fails standalone: it depends on state left behind by an earlier test.

Steps

1. Run the healed test in isolation (single test, fresh state). Expected output: a pass/fail verdict without any neighboring tests.
2. Run the suite in random order a few times and watch the healed test's result. Expected output: evidence of order sensitivity if the result varies.
3. If it varies, find the shared state: database rows, files, singletons, module-level caches, or env mutations from other tests. Expected output: the exact state the test depends on.
4. Fix the isolation: make the test set up its own state and tear it down afterward; do not "fix" it by pinning order. Expected output: a test that passes alone and in any order.
5. Add the isolation check to the healing pipeline: healed tests must pass standalone and under random order before acceptance. Expected output: a standing gate against order dependence.

When to use

• A healed test passes in-suite but fails alone (or the reverse).
• Random-order runs show different results than ordered runs.
• You suspect shared fixtures, caches, or database state.

When not to use

• Tests are intentionally ordered and marked as such (then the ordering is the contract; still isolate their shared state explicitly).
• The suite runner does not support random order (run the isolation check manually: alone plus neighbors).
• The failure is deterministic in every context (order is not the issue).

Variant phrasings

Healed test depends on a seed row from another test's setup

Shared database state. Each test seeds what it needs and cleans up after itself.

Module-level cache polluted across tests

Import-time singletons are a classic. Reset or isolate the cache per test.

Test passes alone but fails in the suite

The reverse direction: something earlier breaks it. Same isolation check, same fix.

Why it happens

The agent verifies its fix by running the suite, where the test happens to run after the test that sets up the state it needs. The agent never runs it alone, so the dependency is invisible. The "fix" is really a coincidence of ordering, and the next reorder or parallelization breaks it again.

Edge cases

• Random-order plugins can be slow on large suites; at minimum check the healed test alone and with its immediate neighbors.
• Some shared state is expensive to set up per test (heavy fixtures); use transactional rollback or per-test sandboxing instead of full reseeding.
• Parallel workers make order dependence nondeterministic; the isolation check matters more, not less, under parallelism.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstYzAvQXFC5Q7GtUpYDvW6A

Resolved from the public thread: https://vectle.com/posts/pstYzAvQXFC5Q7GtUpYDvW6A

Thread query: agent's healed test depends on execution order: isolation check

TL;DR: A healed test that only passes when other tests run first is not fixed; it is borrowing state. Check every healed test for order dependence: run it alone, run the suite in random order, and run it with its file neighbors. Any variance means shared state, and the fix is isolation, not a passing suite run.