the agent quarantined a test for "flake" that was actually catching a deploy config error

TL;DR That "flake" was the only test catching a real deploy config error. Un-quarantine it, fix the config it was flagging, and check what else got skipped that might be load-bearing.

the agent quarantined a test for "flake" that was actually catching a deploy config error

Steps

1. Read what the test actually asserts. Open the quarantined test and find the config value or deploy behavior it checks; that is the thing it was protecting.

Expected: you can state in one sentence what deploy mistake the test guards against.

1. Check the deploy config it was flagging. Compare the current deploy config against what the test expects; the mismatch the test caught is usually still there.

Expected: you find a real config error (wrong env var, bad URL, missing secret reference) that the quarantine hid.

1. Fix the config, not the test. Correct the deploy configuration so the test's assertion holds in the real pipeline.

Expected: the config now matches what the test expects.

1. Un-quarantine the test and rerun it against the fixed config.

Expected: green, with the test back on guard duty.

1. Audit the rest of the quarantine list for load-bearing tests. Any test that validates deploy, config, or infra behavior gets flagged as do-not-quarantine-without-review.

Expected: a protected list so the next agent cannot skip the alarms.

Use this when

• a quarantined test turns out to validate deploy or config behavior
• a deploy failed in a way a skipped test would have caught
• the quarantine reason was timing-based but the test asserts config
• the agent quarantined without reading what the test checks

Not for this skill when

• the test genuinely flakes on timing with no config assertion involved
• there is no deploy pipeline or config under test
• the deploy failure had no test coverage at all (write the test first)
• the test was quarantined for a proven, unrelated flake

Variant phrasings

• "quarantined test was catching a real config error"
• "skipped test would have caught the deploy failure"
• "agent quarantined the test that guards the deploy config"

Why it happens

Agents classify by failure pattern, not by test purpose. A config-validation test that fails intermittently (because the config is wrong in some environments) looks exactly like a flake to a pattern matcher. The agent never asks "what is this test for", so the alarm gets unplugged instead of investigated.

Edge cases

• The config error may have been fixed already while the test stayed quarantined; un-quarantine still matters, otherwise the guard never comes back.
• Mark config-guarding tests explicitly (naming convention or tag) so future triage treats them as high-value.
• If the test is flaky AND load-bearing, fix the flakiness first; a noisy alarm gets ignored, then quarantined.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCOYTiXiLUrwCo0mtWWx7w