agent loop detected: re-running the same failing test without code changes

TL;DR

Give the healing loop a tripwire: record the code hash before every re-run, and if the same test fails twice with an identical hash, stop the loop and escalate. Re-running a failing test without changing anything is not healing, it is burning CI minutes and hiding the real failure.

Error

[09:12:04] test_login.py::test_expired_session FAILED (2.1s)
[09:12:05] healing attempt 1: retrying test, no code changes
[09:12:07] test_login.py::test_expired_session FAILED (2.0s)
[09:12:08] healing attempt 2: retrying test, no code changes
[09:12:10] test_login.py::test_expired_session FAILED (2.1s)
[09:12:11] healing attempt 3: retrying test, no code changes

Steps

1. Before each healing attempt, capture git rev-parse HEAD plus a hash of uncommitted changes (git diff | sha256sum) and store it with the attempt number. Expected: every attempt has a recorded code state.
2. Compare the current code hash to the previous attempt's hash. Expected: if identical and the test failed again, the loop is detected.
3. Set a hard cap: max 3 attempts per test per session, and max 1 attempt with zero code change. Expected: the agent stops itself instead of looping forever.
4. On loop detection, quarantine the test (mark skip with a link to the failure log) and open a ticket for a human. Expected: the suite goes green-ish by quarantine, and the failure is tracked, not retried.
5. Log every attempt (test id, code hash, result) to a JSONL file so you can audit loops later. Expected: grepping the log shows no test with 3 identical-hash failures.

When to use

• An agent keeps re-running a failing test hoping it turns green.
• CI bills show the same test running dozens of times in one session.
• You are building or configuring a test-healing agent loop.

When not to use

• Deliberate flake detection runs (re-running N times on purpose to measure flakiness; different policy, see the flaky-verification skill).
• The agent changed code between runs (that is a real healing attempt, not a loop).

Tool compatibility

• Any coding agent loop (Claude Code, Cursor, custom agents) over pytest, Jest, Vitest, or Go test.
• Shell tooling only: git, sha256sum, JSONL logging.

Variant phrasings

Agent retries the test instead of reading the error

Same loop shape, different rationalization. The tripwire does not care about intent, only about unchanged code plus repeated failure.

Healing attempt with only formatting changes

Whitespace-only diffs change the hash but not behavior. Compare the semantic diff (git diff -w) too, or require the attempt to name the hypothesis it is testing.

Infinite retry on a hung test

Timeouts count as failures for the loop counter. A test that hangs three times is looping, not flaky.

Why it happens

Most agent loops treat "run the tests" as both the diagnostic and the fix. Without a recorded code state, the agent cannot tell attempt 1 from attempt 5, so it retries the same failing command and reports "still investigating". The hash comparison gives the loop a memory.

Edge cases

• The agent changes a config file the hash does not cover: hash the full working tree including untracked config, or record the exact command plus env.
• Legitimate retries after external flakiness (a down dependency): allow one no-change retry, but require the agent to name the external cause in the log.
• Parallel agents healing the same test can each stay under the cap: key the attempt log by test id, not by agent session.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_q6XIbacDOnhmfUgqK68q8A