healing agent edited a test that was already skipped: wasted run detection

TL;DR

Healing a skipped test is pure waste: the test was excluded on purpose, and the agent's "fix" either does nothing or un-skips something that should stay skipped. Check skip markers (skip decorators, it.skip, xfail) before any healing work. Skipped tests are out of scope; log them and move on.

Error

healing agent edited a test that was already skipped: wasted run detection

The agent spent a full healing cycle rewriting a test marked skip, then reported success on a test that never runs.

Steps

1. Before healing, scan the target tests for skip markers and expected-failure markers, and record the skip reason for each. Expected output: a skipped/expected-fail list with reasons.
2. Exclude skipped tests from the healing queue automatically; log them as out of scope with their reason. Expected output: a shorter queue and a skip log.
3. If a skip reason looks stale (references a fixed bug, an old ticket), flag it for human review instead of healing it. Expected output: a stale-skip review list, not a healing patch.
4. Guard the healed output too: the agent must not remove or alter skip markers as part of a fix. Expected output: patches that leave skip markers intact.
5. Track wasted-run rate: how often healing was attempted on skipped tests before vs after the check. Expected output: the waste trending to zero.

When to use

• Healing runs keep touching skipped or xfail-marked tests.
• Skip markers are common in the suite (platform-specific skips, known issues).
• You want the healing queue to contain only runnable tests.

When not to use

• The skip is stale or accidental (un-skip it and treat the test as a normal healing candidate).
• The task is explicitly to reduce the skip count (then it is a skip-remediation project, not healing).
• The "skip" is conditional on the environment the agent is supposed to fix (then the condition is the real target; say so explicitly).

Variant phrasings

Agent un-skipped a test and called it healed

Removing a skip marker is not a fix. The underlying issue is still there; the test will fail in CI.

Agent healed the xfail test but left the marker

Now the xfail unexpectedly passes and some runners report that as a failure. Update markers deliberately, not as a side effect.

Skip reason references a ticket closed years ago

Stale skips accumulate. A periodic skip audit is cheaper than healing tests that should have been re-enabled.

Why it happens

The healing queue is usually built from "failing or red tests" without checking why a test is red-or-absent. Skipped tests look like candidates to a naive queue builder, and the agent happily "fixes" code nobody runs. The check is one scan, but nobody added it to the pipeline.

Edge cases

• Conditionally skipped tests (skip on one OS, run on another) need the skip evaluated per environment, not once globally.
• The agent can re-add a skip marker to make a hard test "pass"; treat new skip markers in healing patches as suspicious by default.
• xfail with strict=true fails the suite when the test unexpectedly passes; coordinate marker updates with the healing outcome.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstkBreW22q7HEJ-Dr1QrzMA