agent healed tests against a stale build artifact: how to detect

TL;DR

Healing tests against a stale build is fiction: the agent "fixes" tests for code that is not what is in the repo. Detect it by stamping every build artifact with its source commit hash and refusing to heal when the artifact's hash does not match the current source. Rebuild first, heal second, always in that order.

Error

agent healed tests against a stale build artifact: how to detect

The agent's fixes passed against yesterday's build, then failed on the fresh build because the source had moved on.

Steps

1. Stamp the artifact: embed the source commit hash (and a build timestamp) into the build output or a sidecar manifest. Expected output: every artifact carries its provenance.
2. Before healing, compare the artifact's hash against the current source hash; mismatch means stale. Expected output: a clear fresh-or-stale verdict.
3. On stale, rebuild from current source and re-verify the hash matches before the agent starts. Expected output: a fresh artifact the healing run can trust.
4. Make rebuild-before-heal the pipeline order, enforced, not suggested. Expected output: no healing run can start on a stale artifact.
5. Log the artifact hash with every healing result so a later "it passed then" claim can be checked. Expected output: provenance attached to every healing report.

When to use

• Healed tests pass, then fail on the next fresh build.
• The agent's environment caches builds aggressively.
• Source and artifact can drift (long-lived agent workspaces, shared build caches).

When not to use

• Interpreted codebases with no build step (nothing to go stale).
• The build is hermetic and content-addressed (staleness is structurally impossible).
• The failure is in the build itself (then fix the build; healing tests is the wrong lane).

Variant phrasings

Agent tested against a cached Docker layer

Layer caches are stale builds in disguise. Bust the cache or verify the hash inside the image.

Shared build cache served another branch's artifact

Cache keys must include the branch and commit; a cache hit on the wrong key is a stale build.

Agent healed, then someone rebuilt, and the fix vanished

The fix was real but validated against the wrong artifact. Re-validate after every rebuild.

Why it happens

Builds are slow, so pipelines cache them, and agents reuse whatever artifact is lying around. Nothing checks that the artifact matches the source the agent is about to "fix" tests for. The agent does good work against the wrong binary, and the truth only appears at the next clean build.

Edge cases

• Timestamps alone are unreliable (clock skew, rebuilds that change nothing); the commit hash is the source of truth.
• Partial rebuilds can mix fresh and stale objects; prefer clean rebuilds for healing validation, or verify per-object hashes.
• The agent can trigger a rebuild but forget to wait for it; the hash check after the build closes that race.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VbEd0jb8v590vdPttvAU2g