got "coverage threshold for branches not met" after the agent quarantined half the suite: how to recover
Recovers from the 'coverage threshold for branches not met' failure caused by a test-healing agent quarantining so many tests that branch coverage collapsed. Shows how to audit the quarantine list, restore stable tests, rerun coverage on the honest suite, and put guardrails on the agent's quarantine behavior. Use when coverage gates fail right after an agent quarantined tests. Not for genuine coverage drops from deleted tests, misconfigured coverage tools, or threshold changes.
TL;DR
Branch coverage fell because the quarantined tests were the ones exercising those branches, not because the code got worse. Audit the agent's quarantine list, un-quarantine every test that is actually stable, rerun coverage on the restored suite, and only then decide whether the threshold itself needs to change. Then cap the agent's quarantine power so it cannot silently gut coverage again.
got "coverage threshold for branches not met" after the agent quarantined half the suite: how to recoverSteps
- Open the agent's quarantine list (the skip file, quarantine config, or the PR that added the skips) and count what was quarantined and when. Expected: you have the full list of skipped tests with dates.
- Audit the list for overreach: tests quarantined after a single failure, tests that never failed at all, and quarantine entries months old that were never re-verified. Expected: you can mark each entry as legit-flaky or wrongly-quarantined.
- Restore the wrongly-quarantined tests: remove their skip flags and run them. Keep quarantined only the ones that fail repeatedly for a real flake reason. Expected: the suite runs its honest test count again.
- Rerun coverage on the restored suite and compare branch coverage to the threshold. Expected: coverage recovers toward its previous level; the remaining gap is real and small.
- Put guardrails on the agent: a maximum quarantine percentage that pages a human, mandatory re-verification before any quarantine, and expiry dates on every quarantine entry. Expected: the agent cannot silently halve the suite again.
Use this when
- The coverage gate fails right after an agent quarantined tests
- The quarantine list is large relative to the suite
- Branch coverage dropped without any code change
Not for this skill when
- Coverage dropped because tests were legitimately deleted
- The coverage tool is misconfigured (wrong include paths, stale reports)
- The threshold was raised deliberately and the code needs more tests
Variant phrasings
- coverage threshold not met after quarantining flaky tests
- agent quarantined too many tests coverage dropped
- recover branch coverage after test quarantine
Why it happens
Coverage measures which branches the RUNNING tests exercise. Quarantining a test removes its coverage contribution, so an aggressive quarantine pass deletes coverage without touching a line of product code. The threshold gate then fails on a coverage number that no longer reflects the code, it reflects the shrunken suite.
Edge cases
- If some quarantined tests are genuinely flaky, fix or rewrite them instead of restoring them blindly; restored flakes will just fail the suite instead of the coverage gate.
- Do not lower the threshold to match the gutted suite. That bakes the coverage hole into the standard and hides it permanently.
- Track quarantine percentage as its own CI metric with an alert; coverage alone tells you too late.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_2hoOS9bm4u6J938Eq8RsRQ