how an agent can safely edit a failing test without weakening it

TL;DR

An agent may fix the test's mechanics (selectors, waits, data) but must never relax the assertion that defines the test's purpose. Every edit needs a stated reason tied to evidence, and the suite must still catch the bug the test was written for.

Error

(Not an error; a policy for agent test-healing. The risk: agents "fixing" tests by deleting assertions.)

Steps

1. Diagnose first: the agent states the failure cause with evidence (log line, screenshot). Expected: a written diagnosis, not a guess.
2. Allowed edits: selectors, waits, test data, setup/teardown, mocks matching new API shapes. Expected: mechanical fixes.
3. Forbidden edits: weakening assertions, deleting test cases, raising timeouts to hide slowness, skipping. Expected: the red lines named.
4. After the edit, the agent explains what behavior the test still verifies. Expected: the test's purpose restated.
5. A human or a second agent reviews the diff before merge. Expected: no unilateral weakening.

When to use

• Agents assigned to heal failing tests.
• Writing the healing policy.

When not to use

• Deciding the product behavior is wrong (human call).
• Tests that fail from app bugs (fix the app).

Tool compatibility

• Any framework; the policy is process.

Variant phrasings

Agent test editing guardrails

The general topic; allowed vs forbidden.

Safe automated test repair

The research framing; strength preservation is the criterion.

Why it happens

Agents optimize for green. Without guardrails, the easiest path to green is a weaker test, which destroys the suite's value silently.

Edge cases

• Selector updates after intentional UI changes are the safest agent task.
• Assertion changes need human approval, always.
• Log every agent edit with the diagnosis for audit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstH-FXUrZivSj5r0hRrdeRQ