agent's quarantine PR also skipped 12 tests that were never failing - how to audit an agent's skip list

TL;DR

Every skipped test must cite the failing run that earned its skip. Diff the quarantine list against real failure history, delete every entry with no evidence, and re-verify the rest on a schedule. A skip list without per-test evidence is just coverage being thrown away.

The exact query

agent's quarantine PR also skipped 12 tests that were never failing  -  how to audit an agent's skip list

Steps

1. Export the full skip list: every test the agent's PR skips, quarantines, or marks expected-fail, with the reason attached to each entry. If entries have no reason, that is already a finding.

Expected: A complete list of skipped tests, each with (or conspicuously without) a justification.

1. For each entry, find the failing run that justifies it: the CI run id, the error, the date. A legitimate skip cites a real failure. The 12 tests that were never failing will have no such run; that is how you identify them.

Expected: Each entry is marked "justified" (failing run cited) or "unjustified" (no failure on record). The unjustified ones are the audit findings.

1. Un-skip everything unjustified immediately and run those tests. They were passing before the agent touched them; confirm they still pass and keep them in the suite.

Expected: The never-failing tests are back in the suite and green. Coverage is restored.

1. For the justified entries, check the justification is still true: re-run each quarantined test now. Tests quarantined months ago may have been fixed by later changes; a skip with no expiry rots.

Expected: Stale quarantines (tests that now pass) are removed. The list shrinks to tests that still genuinely fail.

1. Put the list under governance: every new skip needs a cited failing run, an owner, and an expiry date with automatic re-verification. Review the list monthly and graph its size; a growing list is a quality signal going the wrong way.

Expected: The skip list is small, evidenced, owned, and shrinking over time instead of silently growing.

Use this when

• An agent's quarantine PR skipped tests that were never failing
• Coverage dropped after an agent's change and nobody can explain why
• You inherited a skip list and need to know which entries are legitimate
• You are writing quarantine policy for a test-healing agent

Not for this skill when

• Every skip already cites a failing run, an owner, and an expiry (then the list is governed; audit the underlying failures instead)
• The skips are intentional and documented (deprecated feature, platform-specific, known upstream bug with a link)
• The problem is tests being skipped by developers manually for convenience (that is a culture problem, not an agent-audit problem)

Variant phrasings

how to review a quarantine PR from an agent

Demand per-test evidence: which run failed, with what error. No evidence, no skip.

agent skipped tests that were passing

Un-skip them, run them, and add the evidence rule so it cannot happen again.

quarantine list keeps growing

Put expiries and monthly reviews on it. An unbounded skip list is a slow-motion suite deletion.

Why it happens

Agents quarantine by pattern, not by evidence: one failing test in a file becomes "quarantine the file", a flaky area becomes "skip the directory". Each individual skip feels cheap, but the list compounds silently because nothing forces re-verification. The 12 never-failing tests got swept up as collateral, and since skipped tests do not run, nobody notices they are gone until coverage numbers sag or a regression ships through the hole.

Edge cases

• Skip markers committed directly in test files (skip decorators, focused-test commits) are harder to audit than a central quarantine list. Sweep the codebase for skip markers as part of the audit.
• A test may have been failing only under a condition that no longer exists (old runner, old dependency). Re-running is the only way to know; the historical failure alone does not justify a permanent skip.
• If the agent's skip list lives in CI config rather than the repo, it may not get code review. Move it into the repo or require review on the config change.
• Beware the "temporarily skipped" entry from 6 months ago. Temporary skips without expiries are permanent skips with better PR.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHDvD-T4yiXZ6Dc5bJSZtQ