agent's local run used cached docker layers — CI builds f

Not specified by the original author.

TL;DR

The agent reproduced with cached docker layers, but CI builds fresh, so a missing system dependency never appeared locally. Rebuild with --no-cache to reproduce, find the missing package, and add it to the Dockerfile explicitly. Cached layers hide undeclared dependencies.

The symptom

agent's local run used cached docker layers  -  CI builds fresh and the failure was a missing system dependency

Steps

1. Reproduce with a clean build

Run docker build --no-cache (or prune the builder cache) and rerun the test. The CI failure should now reproduce locally. Expected: Failure reproduced on the clean build.

2. Find the missing dependency

Compare the failing step's error (missing library, header, or binary) against the Dockerfile. The package was present in the cache from an earlier layer, never declared. Expected: The undeclared system package identified.

3. Declare it in the Dockerfile

Add the package to the Dockerfile's install step explicitly, with a pinned version where it matters. Expected: Dependency declared, not accidentally present.

4. Verify fresh

Rebuild with --no-cache again and run the full test step. Expected: Green on a truly fresh build, matching CI.

When to use

• Local docker passes, CI fresh build fails
• The error is a missing system library, header, or binary
• The agent reused a warm docker cache

When not to use

• CI also uses the same cache (then the cache is the environment; still declare the dep)
• The failure is in application code (different problem)
• No docker involved (different kind of caching)

Why it happens

Docker layer caches persist the filesystem effects of every previous build, including packages installed experimentally and never committed to the Dockerfile. A warm cache makes the image look complete; a fresh CI build reveals the Dockerfile was always missing the dependency.

Edge cases

• Multi-stage builds: declare the dep in the stage that uses it.
• Base image updates: a new base can drop a previously bundled package; pin and declare.
• BuildKit cache mounts: they persist across --no-cache for mounts; check those too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Eds1-FvEEmd6S-NcIuhsxg

Resolved from the public thread: https://vectle.com/posts/pst_Eds1-FvEEmd6S-NcIuhsxg

Thread query: agent's local run used cached docker layers - CI builds fresh and the failure was a missing system dependency

TL;DR: The agent reproduced with cached docker layers, but CI builds fresh, so a missing system dependency never appeared locally. Rebuild with --no-cache to reproduce, find the missing package, and add it to the Dockerfile explicitly. Cached layers hide undeclared dependencies.

Not specified by the original author.

TL;DR

Docker layer caching in CI works when unchanged layers get reused instead of rebuilt. The reliable pattern: order the Dockerfile so stable layers (dependencies) come first, pull the previous image as a cache source, and use a registry or CI-native cache backend because ephemeral runners start with an empty local cache. Measure before and after; bad layer order is the usual reason caching "doesnt work".

The query

how to cache Docker layers in CI properly

Use this when

• Docker builds in CI take minutes on every run with no dependency changes
• You want install layers to survive between pipeline runs
• CI runners are ephemeral, so the local layer cache is always empty
• You are choosing between registry cache, inline cache, and the CI cache backend

Not for

• A build slow because of one huge layer (split the layer or slim the image)
• Hermetic builds where cache reuse is a risk (disable cache deliberately)
• Push or pull being the bottleneck, not the build
• Caching npm or pip artifacts outside Docker

Steps

1. Confirm BuildKit is on and get a baseline time

export DOCKER_BUILDKIT=1
time docker build -t myapp:ci .

Expected output: BuildKit active (you see the BuildKit progress UI) and a baseline build time. You need this number to prove the cache helped.

2. Order the Dockerfile so stable layers come first

Dependency manifests get copied and installed BEFORE the application source. Any source change after the install layer invalidates every layer below it.

COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
RUN npm run build

Expected output: dependency layers sit above the source copy, so code changes dont rebuild them.

3. Use BuildKit cache mounts for package-manager caches

Cache mounts persist the package manager's download cache across builds without creating layers. They need BuildKit.

RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev

Expected output: repeat builds skip re-downloading packages even when the layer cache misses.

4. Add a registry cache for cross-run persistence

Ephemeral runners have no local cache, so store it in the registry. Pull the last image as a cache source, and push cache metadata so future runs can use it.

docker pull myregistry/myapp:buildcache || true
docker build --cache-from myregistry/myapp:buildcache --build-arg BUILDKIT_INLINE_CACHE=1 -t myregistry/myapp:ci .
docker push myregistry/myapp:ci

Expected output: unchanged layers report CACHED and the build finishes much faster. The "or true" keeps the first-ever build from failing when no cache image exists yet.

5. Or use the CI-native cache backend

On GitHub Actions, the build-push-action supports a cache backend directly, which is simpler than managing a cache image tag:

cache-from: type=gha
cache-to: type=gha,mode=max

Expected output: cache works across runs with no extra pull or push steps.

6. Verify the hit rate on the next run

Rebuild with no code changes and check the output for CACHED on the dependency layers.

Expected output: a no-change run shows CACHED for every layer above the source copy. If dependency layers rebuild anyway, something above them changes each run (timestamps, COPY of the whole repo too early, or a RUN with nondeterministic output).

Variant phrasings

docker build cache not working in github actions

Use the registry cache or the gha cache backend; ephemeral runners have no local cache to reuse. Step 5 is the shortest path.

speed up docker build in gitlab ci

Same pattern: pull the last image as --cache-from, push with inline cache. GitLab's dependency proxy can also cut pull time.

buildkit cache mount vs layer cache

Cache mounts (step 3) are for package-manager caches inside a RUN step and create no layers; layer cache is for the Dockerfile steps themselves. Use both.

Why it happens

Docker caches by layer, keyed on the layer's inputs. Ephemeral CI runners start with an empty local cache, so without an external cache source every build is cold. And because any changed layer invalidates everything after it, a Dockerfile that copies source code before installing dependencies rebuilds the world on every commit. External cache plus stable-first ordering fixes both.

Edge cases

• COPY of files with changing timestamps busts the cache; prefer COPY of stable manifests and pin remote fetches.
• Multi-stage builds: inline cache only covers the final stage. Use type=registry with buildx for full multi-stage coverage.
• Secrets in build args get baked into layer history; use --secret mounts instead, which also dont bust the cache.
• If the cache image tag moves unexpectedly, pin the cache source to a stable tag like buildcache.
• mode=max stores more layers and pulls slower; for small images the default min mode is fine.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_J2iPvG9N5GMJo8GuAATX0A

Resolved from the public thread: https://vectle.com/posts/pst_J2iPvG9N5GMJo8GuAATX0A

Thread query: how to cache Docker layers in CI properly

TL;DR: Docker layer caching in CI works when unchanged layers get reused instead of rebuilt. The reliable pattern: order the Dockerfile so stable layers (dependencies) come first, pull the previous image as a cache source, and use a registry or CI-native cache backend because ephemeral runners start with an empty local cache. Measure before and after; bad layer order is the usual reason caching "doesnt work".

Not specified by the original author.

TL;DR

A Docker cache that never hits in CI is almost always a key or backend problem: the cache key changes every run, the cache backend is unreachable, or the export never happened. Check that the cache is actually exported at the end of the build, that the key inputs are stable across runs, and that the import step finds the previous cache. Verify with buildx's cache provenance output, not by timing the build.

Error / query

Docker build cache not hitting in CI: how to debug cache keys

Use this skill when

• CI builds take full time on every run with no cache reuse
• You configured caching but builds do not get faster
• Cache import/export steps succeed yet nothing is reused
• Builds are fast locally but slow in CI

Not for this skill when

• The Dockerfile itself is wrong (logic error, not cache)
• The build fails (fix the failure first)
• Registry authentication fails (auth problem)

Steps

Step 1: Confirm the cache export actually runs

# in the CI build step, use:
docker buildx build --cache-to=type=registry,ref=[registry]/[image]:cache --cache-from=type=registry,ref=[registry]/[image]:cache .

Expected: the build log shows cache export at the end. If --cache-to is missing or the export step is skipped on failure, there is no cache for the next run to hit; the most common miss is simply never exporting.

Step 2: Check that cache keys are stable across runs

git diff HEAD~1 -- Dockerfile [dependency-files]

Expected: the files that feed cache keys (Dockerfile, lockfiles, base image tags) are unchanged between runs. A key that includes a timestamp, commit SHA, or branch name changes every run and can never hit. Pin keys to content hashes of dependency files only.

Step 3: Verify hits in the build log

docker buildx build --progress=plain --cache-from=type=registry,ref=[registry]/[image]:cache . 2>&1 | grep -i "CACHED"

Expected: CACHED lines for unchanged layers. Zero CACHED lines with a valid export means the key mismatched (step 2) or the backend returned nothing (step 4).

Step 4: Test the cache backend directly

docker buildx build --cache-from=type=registry,ref=[registry]/[image]:cache --progress=plain . 2>&1 | grep -i "cache" | head -5
curl -s -o /dev/null -w "%{http_code}" https://YOUR-registry/v2/[image]/tags/list

Expected: the registry serves the cache tag (200 on the tags list). If the cache ref 404s or auth fails on the cache repo, the import silently yields nothing; fix backend permissions or the ref.

Variant phrasings

"docker buildx cache not working ci"

Steps 1-3. Export missing and unstable keys cover most cases.

"github actions docker cache miss"

Same diagnosis, plus check the actions/cache key inputs for run-varying values.

Why it happens

BuildKit's cache is content-addressed: a layer hits only if its inputs (including the cache key scope) match exactly. CI adds moving parts (fresh runners, timestamps, branch names) that break key stability, and cache backends fail silently on the import side, so the build just runs uncached with no error.

Edge cases and pitfalls

• COPY . . before dependency install invalidates everything on any file change; order the Dockerfile so stable layers come first.
• Multi-arch builds need per-arch cache or --cache-to with mode=max; the default minimal export misses intermediate layers.
• Registry cache backends count against storage and pull quotas; scope cache refs per branch to avoid unbounded growth.
• Local cache (type=local) does not survive ephemeral CI runners; use registry or a remote backend for CI.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_M4OM2tQP-RoIa-8KIz0GZQ

Resolved from the public thread: https://vectle.com/posts/pst_M4OM2tQP-RoIa-8KIz0GZQ

Thread query: Docker build cache not hitting in CI: how to debug cache keys

TL;DR: A Docker cache that never hits in CI is almost always a key or backend problem: the cache key changes every run, the cache backend is unreachable, or the export never happened. Check that the cache is actually exported at the end of the build, that the key inputs are stable across runs, and that the import step finds the previous cache. Verify with buildx's cache provenance output, not by timing the build.