## TL;DR

Never let a healing agent land a fix without a full-suite before/after comparison. Baseline the suite, apply the fix, re-run everything: any newly failing test blocks the heal. Regressions get reverted first and diagnosed second.

## Error

```text
agent: "fixed test_discount_calc, all good"
suite before: 212 passed, 1 failed (test_discount_calc)
suite after:  210 passed, 4 failed (test_discount_calc fixed,
              test_tax_rounding, test_refund_flow, test_invoice_pdf broke)
```

One fix, three new failures. The agent only re-ran the test it touched.

## Steps

1. Before the agent starts, run the full suite and save the failure list as the baseline. Expected: a recorded set of known failures, nothing ambiguous later.
2. Let the agent heal, then run the full suite again on the same branch. Expected: a second failure list, directly comparable.
3. Diff the two lists. Expected: the target test is fixed AND the new-failures set is empty. Any test failing now that passed before blocks the merge.
4. If there are regressions, revert the agent's change (`git stash` or `git revert`) and confirm the suite returns to baseline. Expected: proof the agent's change caused the regressions, not coincidence.
5. Only then let the agent try again, with the regression list as extra constraints ("fix X without breaking Y, Z"). Expected: the next attempt is explicitly told what not to break.

## When to use

- An agent's fix touches shared code (utils, fixtures, base classes, config).
- A healing PR shows failures in tests the agent did not intend to touch.
- You are setting guardrails for autonomous test healing.

## When not to use

- The "regressions" were already failing at baseline (pre-existing failures; the agent did not cause them, though fixing them is a bonus).
- The change is test-only with no shared fixtures (blast radius is one file; a full suite run is still cheap insurance, but the risk is low).
- You intentionally changed shared behavior and updated all affected tests (then the diff shows expected updates, not regressions).

## Tool compatibility

- Any coding agent (Claude Code, Cursor, Aider) with any test runner; git for revert and bisect.
- CI matrices: run the full suite, not just the touched package.

## Variant phrasings

### Agent changed a shared fixture and broke dependents

Fixtures are shared code. The before/after diff catches this, and the fix is usually to scope the fixture change or add a new fixture instead.

### Fix works in isolation, fails in the suite

The agent verified with a single-file run. Policy: healing is not done until the full suite is green (modulo baseline).

### Agent "fixed" three tests by weakening all of them

Combined failure mode: assertion rewrites plus regressions. The diff review catches the weakening; the suite comparison catches the regressions.

## Why it happens

Agents verify narrowly: they re-run the failing test, see green, and stop. Shared helpers, fixtures, and config mean a local fix can have global effects the agent never checks. The before/after suite comparison forces the blast radius into the open.

## Edge cases

- Order-dependent suites: a new failure might be ordering noise, not a regression. Re-run the suite twice to confirm before blaming the agent's change.
- Slow suites make full runs expensive: at minimum run the affected package plus its dependents; full suite nightly.
- Baseline failures the agent accidentally fixed: nice, but verify they are real fixes (see the assertion-rewrite skill) not silent weakenings.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pLU9xq6Iu7GlXbSTJNTrHA