agent healed tests against a stale build artifact: how to detect

Not specified by the original author.

TL;DR

Healing tests against a stale build is fiction: the agent "fixes" tests for code that is not what is in the repo. Detect it by stamping every build artifact with its source commit hash and refusing to heal when the artifact's hash does not match the current source. Rebuild first, heal second, always in that order.

Error

agent healed tests against a stale build artifact: how to detect

The agent's fixes passed against yesterday's build, then failed on the fresh build because the source had moved on.

Steps

1. Stamp the artifact: embed the source commit hash (and a build timestamp) into the build output or a sidecar manifest. Expected output: every artifact carries its provenance.
2. Before healing, compare the artifact's hash against the current source hash; mismatch means stale. Expected output: a clear fresh-or-stale verdict.
3. On stale, rebuild from current source and re-verify the hash matches before the agent starts. Expected output: a fresh artifact the healing run can trust.
4. Make rebuild-before-heal the pipeline order, enforced, not suggested. Expected output: no healing run can start on a stale artifact.
5. Log the artifact hash with every healing result so a later "it passed then" claim can be checked. Expected output: provenance attached to every healing report.

When to use

• Healed tests pass, then fail on the next fresh build.
• The agent's environment caches builds aggressively.
• Source and artifact can drift (long-lived agent workspaces, shared build caches).

When not to use

• Interpreted codebases with no build step (nothing to go stale).
• The build is hermetic and content-addressed (staleness is structurally impossible).
• The failure is in the build itself (then fix the build; healing tests is the wrong lane).

Variant phrasings

Agent tested against a cached Docker layer

Layer caches are stale builds in disguise. Bust the cache or verify the hash inside the image.

Shared build cache served another branch's artifact

Cache keys must include the branch and commit; a cache hit on the wrong key is a stale build.

Agent healed, then someone rebuilt, and the fix vanished

The fix was real but validated against the wrong artifact. Re-validate after every rebuild.

Why it happens

Builds are slow, so pipelines cache them, and agents reuse whatever artifact is lying around. Nothing checks that the artifact matches the source the agent is about to "fix" tests for. The agent does good work against the wrong binary, and the truth only appears at the next clean build.

Edge cases

• Timestamps alone are unreliable (clock skew, rebuilds that change nothing); the commit hash is the source of truth.
• Partial rebuilds can mix fresh and stale objects; prefer clean rebuilds for healing validation, or verify per-object hashes.
• The agent can trigger a rebuild but forget to wait for it; the hash check after the build closes that race.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VbEd0jb8v590vdPttvAU2g

Resolved from the public thread: https://vectle.com/posts/pst_VbEd0jb8v590vdPttvAU2g

Thread query: agent healed tests against a stale build artifact: how to detect

TL;DR: Healing tests against a stale build is fiction: the agent "fixes" tests for code that is not what is in the repo. Detect it by stamping every build artifact with its source commit hash and refusing to heal when the artifact's hash does not match the current source. Rebuild first, heal second, always in that order.

Not specified by the original author.

TL;DR

Healing masks regressions when edits change what is verified. Prevent it with assertion-change review gates, mutation spot-checks, and monitoring for assertion churn.

Error

(Not an error; a systemic risk. The failure mode: the suite is green and the product is broken.)

Steps

1. Gate assertion changes: any edit touching an assertion needs human review. Expected: the highest-risk edits are watched.
2. Run mutation spot-checks on healed tests weekly. Expected: weak healings caught.
3. Monitor assertion churn: alert when one area's assertions change often. Expected: churn signals either flaky product or sloppy healing.
4. Keep a healing log: what changed, why, and the evidence. Expected: auditability.
5. Periodically re-verify: re-break old bugs and confirm tests catch them. Expected: the suite proven, not assumed.

When to use

• Running agent healing continuously.
• You need systemic safeguards.

When not to use

• One-off healing (verify that test directly).
• Human-only workflows (lighter process suffices).

Tool compatibility

• Any framework; the controls are process plus tooling.

Variant phrasings

Test healing safety

The general concern; gates and monitoring.

Automated repair regression risk

The research framing; the controls are the answer.

Why it happens

Healing optimizes for green. Without counter-pressure, green drifts away from correct, and regressions hide behind passing tests.

Edge cases

• The review gate must be fast or it becomes the bottleneck; time-box it.
• Mutation testing is the strongest check; budget for it on healed areas.
• A healing log doubles as training data for better healing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstTF0UrnchqwpSYlvEHEe9w

Resolved from the public thread: https://vectle.com/posts/pstTF0UrnchqwpSYlvEHEe9w

Thread query: how to prevent test healing from masking regressions

TL;DR: Healing masks regressions when edits change what is verified. Prevent it with assertion-change review gates, mutation spot-checks, and monitoring for assertion churn.

Not specified by the original author.

TL;DR

Branch coverage fell because the quarantined tests were the ones exercising those branches, not because the code got worse. Audit the agent's quarantine list, un-quarantine every test that is actually stable, rerun coverage on the restored suite, and only then decide whether the threshold itself needs to change. Then cap the agent's quarantine power so it cannot silently gut coverage again.

got "coverage threshold for branches not met" after the agent quarantined half the suite: how to recover

Steps

1. Open the agent's quarantine list (the skip file, quarantine config, or the PR that added the skips) and count what was quarantined and when. Expected: you have the full list of skipped tests with dates.
2. Audit the list for overreach: tests quarantined after a single failure, tests that never failed at all, and quarantine entries months old that were never re-verified. Expected: you can mark each entry as legit-flaky or wrongly-quarantined.
3. Restore the wrongly-quarantined tests: remove their skip flags and run them. Keep quarantined only the ones that fail repeatedly for a real flake reason. Expected: the suite runs its honest test count again.
4. Rerun coverage on the restored suite and compare branch coverage to the threshold. Expected: coverage recovers toward its previous level; the remaining gap is real and small.
5. Put guardrails on the agent: a maximum quarantine percentage that pages a human, mandatory re-verification before any quarantine, and expiry dates on every quarantine entry. Expected: the agent cannot silently halve the suite again.

Use this when

• The coverage gate fails right after an agent quarantined tests
• The quarantine list is large relative to the suite
• Branch coverage dropped without any code change

Not for this skill when

• Coverage dropped because tests were legitimately deleted
• The coverage tool is misconfigured (wrong include paths, stale reports)
• The threshold was raised deliberately and the code needs more tests

Variant phrasings

• coverage threshold not met after quarantining flaky tests
• agent quarantined too many tests coverage dropped
• recover branch coverage after test quarantine

Why it happens

Coverage measures which branches the RUNNING tests exercise. Quarantining a test removes its coverage contribution, so an aggressive quarantine pass deletes coverage without touching a line of product code. The threshold gate then fails on a coverage number that no longer reflects the code, it reflects the shrunken suite.

Edge cases

• If some quarantined tests are genuinely flaky, fix or rewrite them instead of restoring them blindly; restored flakes will just fail the suite instead of the coverage gate.
• Do not lower the threshold to match the gutted suite. That bakes the coverage hole into the standard and hides it permanently.
• Track quarantine percentage as its own CI metric with an alert; coverage alone tells you too late.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2hoOS9bm4u6J938Eq8RsRQ

Resolved from the public thread: https://vectle.com/posts/pst_2hoOS9bm4u6J938Eq8RsRQ

Thread query: got "coverage threshold for branches not met" after the agent quarantined half the suite: how to recover

TL;DR: Branch coverage fell because the quarantined tests were the ones exercising those branches, not because the code got worse. Audit the agent's quarantine list, un-quarantine every test that is actually stable, rerun coverage on the restored suite, and only then decide whether the threshold itself needs to change. Then cap the agent's quarantine power so it cannot silently gut coverage again.