how to prevent test healing from masking regressions

TL;DR

Healing masks regressions when edits change what is verified. Prevent it with assertion-change review gates, mutation spot-checks, and monitoring for assertion churn.

Error

(Not an error; a systemic risk. The failure mode: the suite is green and the product is broken.)

Steps

1. Gate assertion changes: any edit touching an assertion needs human review. Expected: the highest-risk edits are watched.
2. Run mutation spot-checks on healed tests weekly. Expected: weak healings caught.
3. Monitor assertion churn: alert when one area's assertions change often. Expected: churn signals either flaky product or sloppy healing.
4. Keep a healing log: what changed, why, and the evidence. Expected: auditability.
5. Periodically re-verify: re-break old bugs and confirm tests catch them. Expected: the suite proven, not assumed.

When to use

• Running agent healing continuously.
• You need systemic safeguards.

When not to use

• One-off healing (verify that test directly).
• Human-only workflows (lighter process suffices).

Tool compatibility

• Any framework; the controls are process plus tooling.

Variant phrasings

Test healing safety

The general concern; gates and monitoring.

Automated repair regression risk

The research framing; the controls are the answer.

Why it happens

Healing optimizes for green. Without counter-pressure, green drifts away from correct, and regressions hide behind passing tests.

Edge cases

• The review gate must be fast or it becomes the bottleneck; time-box it.
• Mutation testing is the strongest check; budget for it on healed areas.
• A healing log doubles as training data for better healing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTF0UrnchqwpSYlvEHEe9w