test healing policy: what agents are allowed to change

TL;DR

Write an explicit allowlist: agents may update selectors, waits, test data, and mocks to match intentional app changes. They may not change assertions, delete tests, or alter timeouts without approval. Everything is logged.

Error

(Not an error; a policy document. The need: agents healing tests without boundaries.)

Steps

1. List allowed changes: selectors, waits, fixtures, mocks, setup/teardown matching new APIs. Expected: the mechanical set.
2. List forbidden changes: assertions, test deletion, skips, timeout increases, production code. Expected: the red lines.
3. Require evidence: every healing references the failure log and the app change. Expected: no unexplained edits.
4. Require verification: the healed test must fail on deliberately broken code. Expected: strength proven.
5. Review and log: human spot-checks plus a full audit log. Expected: accountability.

When to use

• Writing the team test-healing policy.
• Onboarding agents to test maintenance.

When not to use

• Human-only test maintenance (simpler norms suffice).
• Emergency breakage (fix first, document after).

Tool compatibility

• Any framework; the policy is process.

Variant phrasings

Agent test maintenance permissions

The access-control framing.

Automated test repair policy

The formal name; allowlist plus verification.

Why it happens

Agents need boundaries to be useful. An explicit policy turns "be careful" into checkable rules.

Edge cases

• Revisit the policy quarterly; healing capabilities improve.
• Different risk levels per suite area (payments vs marketing pages).
• The log is also the dataset for improving the healer.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstKoT-4J-FOghTkhjEWOB-g