agent's fix passed in isolation but failed in the full suite - a module-level cache leaked between tests

TL;DR

The fix passed in isolation but the full suite failed because a module-level cache leaked state between tests. Find the cache (module globals, singletons, lru_cache), clear it in setup or teardown, and never accept a healing patch without running the affected suite subset, not just the single test.

The symptom

agent's fix passed in isolation but failed in the full suite  -  a module-level cache leaked between tests

Steps

1. Reproduce the suite-order failure

Run the healed test together with the tests that run before it in the suite (or the full file). Confirm it fails in company but passes alone. Expected: Isolated pass, suite failure: order-dependent state confirmed.

2. Find the leaked cache

Look for module-level dicts, singletons, functools.lru_cache, and class-level registries in the code under test. Add a debug print of cache size in setup to catch the leak red-handed. Expected: The specific cache object identified, holding entries from earlier tests.

3. Clear it in setup/teardown

Add cache clearing to the test setup (or an autouse fixture): call cache_clear(), reset the singleton, or reimport the module. The fix should make the cache lifecycle explicit. Expected: The healed test passes both alone and in the full suite.

4. Require subset runs for every heal

Change the healer's acceptance gate: a fix must pass the target test plus its file (or the tests sharing the module) before it is accepted. Single-test green is not sufficient. Expected: No heal is accepted on isolated evidence alone.

When to use

• Healed test passes alone but fails in the suite
• Failures depend on which tests ran earlier
• Module-level caches, singletons, or registries exist in the code under test

When not to use

• The test fails in isolation too (different problem)
• The ordering issue is in the test code itself (fix the test's assumptions)
• The suite is intentionally order-dependent (rare; document it)

Why it happens

Module-level state persists for the whole test process. In isolation the cache starts empty and the test passes; in the suite, earlier tests populate it and the healed test sees stale entries. The agent validated against the easy case and missed the real one.

Edge cases

• lru_cache on methods: clear via the underlying function attribute.
• Caches populated at import time: reimporting the module in setup may be needed.
• Thread-local caches: clear per test, not just per process.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_o-PEYHWg8T8r9KJKg7Ze5g