how to use BuildKit cache mounts in CI

TL;DR

BuildKit cache mounts (RUN --mount=type=cache) give your build a persistent directory for dependency caches that survives across builds but never lands in the image. Mount the package manager's cache dir (npm, pip, Go modules), and installs become incremental instead of from-scratch. In CI, pair it with a remote cache backend or a warm builder so the mount is not empty on every fresh runner.

Error / query

how to use BuildKit cache mounts in CI

Use this skill when

• Builds re-download all dependencies on every CI run
• You want caches shared across builds without bloating layers
• Layer caching invalidates too aggressively
• Speeding up npm, pip, Go, or Maven installs in Docker builds

Not for this skill when

• You need the cache to ship inside the image (it never does)
• The build failure is in Dockerfile logic (not a cache problem)
• Runners are fully ephemeral with no shared backend (mount starts empty)

Steps

Step 1: Enable BuildKit and add a cache mount

# syntax=docker/dockerfile:1
RUN --mount=type=cache,target=/root/.npm npm ci --no-audit --no-fund

Expected: the build succeeds and the npm cache persists in the mount across builds on the same builder. The # syntax line enables the BuildKit frontend that understands --mount.

Step 2: Mount the right directory per package manager

# Python
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
# Go
RUN --mount=type=cache,target=/go/pkg/mod go build ./...
# Maven
RUN --mount=type=cache,target=/root/.m2 mvn -B package -DskipTests

Expected: each tool's download cache hits on repeat builds. Target the tool's actual cache dir; mounting the wrong path silently caches nothing.

Step 3: Share mounts across parallel stages safely

RUN --mount=type=cache,target=/root/.npm,sharing=locked npm ci

Expected: sharing=locked (default) serializes concurrent writers, sharing=shared allows concurrent reads. For CI builds with parallel stages writing the same cache, locked avoids corruption.

Step 4: Persist the cache across ephemeral CI runners

docker buildx create --name ci-builder --driver docker-container --driver-opt network=host --use
docker buildx build --builder ci-builder .

Expected: a persistent builder keeps mounts warm between runs. On fully ephemeral runners, back the builder with remote cache (--cache-to type=registry) so mounts repopulate; otherwise every run starts cold and mounts buy nothing.

Variant phrasings

"docker buildx cache mount npm"

Step 1 is the pattern; step 2 has the per-tool paths.

"buildkit cache mount ci ephemeral runners"

Step 4. Mounts are builder-local; ephemeral runners need a persistent builder or remote cache backend.

Why it happens

Layer caching is all-or-nothing per instruction: any change above the install layer invalidates the whole dependency download. Cache mounts decouple the download cache from the layer graph, so dependency archives persist even when surrounding layers rebuild. The mount is invisible to the final image, so there is no size or security cost.

Edge cases and pitfalls

• Cache mounts do not work with the classic (non-BuildKit) builder; the # syntax line or DOCKER_BUILDKIT=1 is required.
• Mount contents are not part of the build context hash; a poisoned mount can produce stale builds. Bust with --no-cache on the mount stage when dependencies behave oddly.
• Different base images use different home dirs; /root/.npm is wrong for non-root users. Match the target to the build user.
• Mounts are not shared between builders; a matrix of builders each warms its own mount.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstcNRLkSHYkhKk6Ai5VxaVw