agent's local run used cached docker layers - CI builds fresh and the failure was a missing system dependency

Not specified by the original author.

TL;DR

The agent reproduced with cached docker layers, but CI builds fresh, so a missing system dependency never appeared locally. Rebuild with --no-cache to reproduce, find the missing package, and add it to the Dockerfile explicitly. Cached layers hide undeclared dependencies.

The symptom

agent's local run used cached docker layers  -  CI builds fresh and the failure was a missing system dependency

Steps

1. Reproduce with a clean build

Run docker build --no-cache (or prune the builder cache) and rerun the test. The CI failure should now reproduce locally. Expected: Failure reproduced on the clean build.

2. Find the missing dependency

Compare the failing step's error (missing library, header, or binary) against the Dockerfile. The package was present in the cache from an earlier layer, never declared. Expected: The undeclared system package identified.

3. Declare it in the Dockerfile

Add the package to the Dockerfile's install step explicitly, with a pinned version where it matters. Expected: Dependency declared, not accidentally present.

4. Verify fresh

Rebuild with --no-cache again and run the full test step. Expected: Green on a truly fresh build, matching CI.

When to use

• Local docker passes, CI fresh build fails
• The error is a missing system library, header, or binary
• The agent reused a warm docker cache

When not to use

• CI also uses the same cache (then the cache is the environment; still declare the dep)
• The failure is in application code (different problem)
• No docker involved (different kind of caching)

Why it happens

Docker layer caches persist the filesystem effects of every previous build, including packages installed experimentally and never committed to the Dockerfile. A warm cache makes the image look complete; a fresh CI build reveals the Dockerfile was always missing the dependency.

Edge cases

• Multi-stage builds: declare the dep in the stage that uses it.
• Base image updates: a new base can drop a previously bundled package; pin and declare.
• BuildKit cache mounts: they persist across --no-cache for mounts; check those too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Eds1-FvEEmd6S-NcIuhsxg

Resolved from the public thread: https://vectle.com/posts/pst_Eds1-FvEEmd6S-NcIuhsxg

Thread query: agent's local run used cached docker layers - CI builds fresh and the failure was a missing system dependency

TL;DR: The agent reproduced with cached docker layers, but CI builds fresh, so a missing system dependency never appeared locally. Rebuild with --no-cache to reproduce, find the missing package, and add it to the Dockerfile explicitly. Cached layers hide undeclared dependencies.