how to verify a healed test actually tests the right thing

TL;DR

After healing, prove the test still guards its behavior: mutate the app code the test covers and confirm the test fails, then restore. A healed test that passes on broken code is worse than no test.

Error

(Not an error; a verification practice. The risk: a "fixed" test that verifies nothing.)

Steps

1. State what the test is supposed to catch, in one sentence. Expected: the intent explicit.
2. Introduce the bug deliberately (change the app code the test covers). Expected: a controlled breakage.
3. Run the healed test. Expected: it FAILS on the broken code.
4. Restore the app code. Expected: the test passes again.
5. If the test passed on broken code, the healing weakened it; rewrite the assertions. Expected: only strong tests ship.

When to use

• After any test edit, human or agent.
• Reviewing healed tests.

When not to use

• New tests (write them strong from the start).
• Tests with no clear intent (clarify first).

Tool compatibility

• Any framework; manual mutation or Stryker/PIT.

Variant phrasings

Mutation testing for healed tests

The automated form; Stryker and PIT do this systematically.

Does this test still work

The plain question; break-the-code is the answer.

Why it happens

Edits drift tests away from their intent. The only proof a test works is watching it fail for the right reason.

Edge cases

• Some tests guard absence of behavior; mutate by adding the bad behavior.
• Mutation testing everything is slow; target healed tests specifically.
• Record the verification; it is the healing's acceptance proof.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst210hUUr3kmqvkN9ALC3LA