test-healing agent marked a passing-on-retry test as flaky - the retry hit a cached response and never exercised the...

TL;DR

The test was not flaky; the retry passed because it got a cached response and never ran the code under test. Before marking anything flaky, force the retry to bypass caches (fresh fixtures, cache-busting headers or keys, cleared HTTP cache) and confirm the code path actually executed. Only retries that genuinely exercise the code count as evidence.

The symptom

test-healing agent marked a passing-on-retry test as flaky  -  the retry hit a cached response and never exercised the code

Steps

1. Check whether the retry was cache-served

Look at the retry's logs for cache hits: HTTP 304s, cache-hit headers, fixture reuse, or mock shortcuts. Compare the retry's execution trace against a known-good run. Expected: Evidence the retry skipped the code path (cache hit markers in the trace).

2. Rerun with caches bypassed

Rerun the test with caches disabled or busted: clear the HTTP cache, use unique keys per run, disable response caching in the test client, and rebuild fixtures from scratch. Expected: The rerun actually executes the code under test; cache-hit markers are gone.

3. Classify based on the clean rerun

If the cache-busted rerun fails, the test is genuinely broken, not flaky. If it passes consistently across several cache-busted runs, then consider flakiness. Expected: A classification backed by reruns that provably ran the code.

4. Add a cache-busting rule to the healer

Teach the test-healing agent: a retry only counts as flaky-evidence if the run's trace shows the code path executed. Cache-served passes are inconclusive, never exonerating. Expected: Future retries automatically bypass caches before classification.

When to use

• A test was marked flaky after a single passing retry
• Retry logs show cache hits, 304s, or reused fixtures
• The test fails again the next time caches are cold

When not to use

• The product has a genuine caching bug (fix the product, not the classification)
• The test is flaky for other proven reasons (timing, ordering)
• Retries already bypass caches (look elsewhere)

Why it happens

Healing agents treat 'failed then passed' as the definition of flaky. But a retry that hits a warm cache never executes the changed code, so its pass is meaningless. The agent records a false flaky verdict, the real failure stays hidden, and the test gets quarantined or ignored instead of fixed.

Edge cases

• Layered caches (CDN plus app cache plus fixture cache): bust all of them, not just one.
• Tests that intentionally test caching behavior: exempt them from the cache-busting rule.
• Time-based cache expiry can make a retry pass 'honestly' on a later run; use fixed cache-busting instead of waiting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_W20y2UvDcvRoqbJiBMVLjQ