test healing: when to update the test vs file a bug

TL;DR

Update the test when the app changed intentionally and the test encodes the old behavior; file a bug when the app behavior is wrong. The deciding evidence is intent: specs, designs, and PR descriptions.

Error

(Not an error; a triage decision. The cost of getting it wrong: tests updated to enshrine bugs.)

Steps

1. Read the failure and the recent app changes (PRs, designs). Expected: the intent behind the behavior.
2. If the change was intentional (new copy, moved button), update the test to the new intended behavior. Expected: test matches intent.
3. If the behavior looks unintended (wrong data, crash, regression), file a bug with the test as the reproducer. Expected: the app gets fixed.
4. If intent is unclear, ask the owning team; do not guess. Expected: no unilateral behavior decisions.
5. Record the decision with the evidence link. Expected: auditable triage.

When to use

• Triaging any failing test.
• Agent triage workflows.

When not to use

• The failure is clearly mechanical (selector rot).
• You own both the app change and the test (just update).

Tool compatibility

• Any framework; the decision is process.

Variant phrasings

Is the test wrong or the app wrong

The core question; intent decides.

Test failure triage

The general workflow; this decision is step one.

Why it happens

Tests encode expected behavior. When behavior changes, either the expectation or the behavior is wrong, and only intent distinguishes them.

Edge cases

• Agents must not decide intent alone; escalate ambiguity.
• Flaky failures are neither; fix the flake first, then triage.
• Document the decision; future triagers need the precedent.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uTPjB6R1tjJutbehvwKVOQ