agent's local run used cached docker layers - CI builds fresh and the failure was a missing system dependency
Fixes unreproducible failures caused by stale docker layer caches. Use when the agent's local docker run passes but CI's fresh build fails. Covers rebuilding without cache, finding the missing system dependency, and pinning it in the Dockerfile. Not for application-code failures.
TL;DR
The agent reproduced with cached docker layers, but CI builds fresh, so a missing system dependency never appeared locally. Rebuild with --no-cache to reproduce, find the missing package, and add it to the Dockerfile explicitly. Cached layers hide undeclared dependencies.
The symptom
agent's local run used cached docker layers - CI builds fresh and the failure was a missing system dependencySteps
1. Reproduce with a clean build
Run docker build --no-cache (or prune the builder cache) and rerun the test. The CI failure should now reproduce locally. Expected: Failure reproduced on the clean build.
2. Find the missing dependency
Compare the failing step's error (missing library, header, or binary) against the Dockerfile. The package was present in the cache from an earlier layer, never declared. Expected: The undeclared system package identified.
3. Declare it in the Dockerfile
Add the package to the Dockerfile's install step explicitly, with a pinned version where it matters. Expected: Dependency declared, not accidentally present.
4. Verify fresh
Rebuild with --no-cache again and run the full test step. Expected: Green on a truly fresh build, matching CI.
When to use
- Local docker passes, CI fresh build fails
- The error is a missing system library, header, or binary
- The agent reused a warm docker cache
When not to use
- CI also uses the same cache (then the cache is the environment; still declare the dep)
- The failure is in application code (different problem)
- No docker involved (different kind of caching)
Why it happens
Docker layer caches persist the filesystem effects of every previous build, including packages installed experimentally and never committed to the Dockerfile. A warm cache makes the image look complete; a fresh CI build reveals the Dockerfile was always missing the dependency.
Edge cases
- Multi-stage builds: declare the dep in the stage that uses it.
- Base image updates: a new base can drop a previously bundled package; pin and declare.
- BuildKit cache mounts: they persist across --no-cache for mounts; check those too.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_Eds1-FvEEmd6S-NcIuhsxg