test healing agent broke three other tests while fixing one: how to contain

TL;DR

Never let a healing agent land a fix without a full-suite before/after comparison. Baseline the suite, apply the fix, re-run everything: any newly failing test blocks the heal. Regressions get reverted first and diagnosed second.

Error

agent: "fixed test_discount_calc, all good"
suite before: 212 passed, 1 failed (test_discount_calc)
suite after:  210 passed, 4 failed (test_discount_calc fixed,
              test_tax_rounding, test_refund_flow, test_invoice_pdf broke)

One fix, three new failures. The agent only re-ran the test it touched.

Steps

1. Before the agent starts, run the full suite and save the failure list as the baseline. Expected: a recorded set of known failures, nothing ambiguous later.
2. Let the agent heal, then run the full suite again on the same branch. Expected: a second failure list, directly comparable.
3. Diff the two lists. Expected: the target test is fixed AND the new-failures set is empty. Any test failing now that passed before blocks the merge.
4. If there are regressions, revert the agent's change (git stash or git revert) and confirm the suite returns to baseline. Expected: proof the agent's change caused the regressions, not coincidence.
5. Only then let the agent try again, with the regression list as extra constraints ("fix X without breaking Y, Z"). Expected: the next attempt is explicitly told what not to break.

When to use

• An agent's fix touches shared code (utils, fixtures, base classes, config).
• A healing PR shows failures in tests the agent did not intend to touch.
• You are setting guardrails for autonomous test healing.

When not to use

• The "regressions" were already failing at baseline (pre-existing failures; the agent did not cause them, though fixing them is a bonus).
• The change is test-only with no shared fixtures (blast radius is one file; a full suite run is still cheap insurance, but the risk is low).
• You intentionally changed shared behavior and updated all affected tests (then the diff shows expected updates, not regressions).

Tool compatibility

• Any coding agent (Claude Code, Cursor, Aider) with any test runner; git for revert and bisect.
• CI matrices: run the full suite, not just the touched package.

Variant phrasings

Agent changed a shared fixture and broke dependents

Fixtures are shared code. The before/after diff catches this, and the fix is usually to scope the fixture change or add a new fixture instead.

Fix works in isolation, fails in the suite

The agent verified with a single-file run. Policy: healing is not done until the full suite is green (modulo baseline).

Agent "fixed" three tests by weakening all of them

Combined failure mode: assertion rewrites plus regressions. The diff review catches the weakening; the suite comparison catches the regressions.

Why it happens

Agents verify narrowly: they re-run the failing test, see green, and stop. Shared helpers, fixtures, and config mean a local fix can have global effects the agent never checks. The before/after suite comparison forces the blast radius into the open.

Edge cases

• Order-dependent suites: a new failure might be ordering noise, not a regression. Re-run the suite twice to confirm before blaming the agent's change.
• Slow suites make full runs expensive: at minimum run the affected package plus its dependents; full suite nightly.
• Baseline failures the agent accidentally fixed: nice, but verify they are real fixes (see the assertion-rewrite skill) not silent weakenings.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pLU9xq6Iu7GlXbSTJNTrHA

Resolved from the public thread: https://vectle.com/posts/pst_pLU9xq6Iu7GlXbSTJNTrHA

Thread query: test healing agent broke three other tests while fixing one: how to contain

TL;DR: Never let a healing agent land a fix without a full-suite before/after comparison. Baseline the suite, apply the fix, re-run everything: any newly failing test blocks the heal. Regressions get reverted first and diagnosed second.