how to cache test dependencies in CI

Not specified by the original author.

TL;DR

Cache what is expensive to fetch: package installs, browser binaries, and build outputs. Key caches on lockfiles and tool versions, and verify the cache actually hits.

Error

(Not an error; a speed practice. Symptom: CI spends 5 minutes installing before testing.)

Steps

1. Cache the package manager store keyed on the lockfile hash. Expected: installs become seconds on cache hit.
2. Cache browser binaries keyed on the framework version. Expected: no downloads on warm cache.
3. Cache build outputs if tests need a build. Expected: incremental builds.
4. Verify hit rates in CI logs. Expected: you confirm the cache works, not just exists.
5. Invalidate deliberately on version bumps. Expected: no stale-cache mysteries.

When to use

• Slow CI installs.
• Any repeated CI run.

When not to use

• Cache debugging (sometimes the cache IS the bug; be ready to bust it).
• Tiny projects where install is seconds.

Tool compatibility

• Any CI cache; npm/pnpm/yarn; Playwright/Cypress browser caches.

Variant phrasings

CI cache node_modules

The most common cache; lockfile-keyed.

Cache Playwright browsers in CI

The browser-specific case; version-keyed.

Why it happens

Installs and downloads are pure overhead repeated every run. Caches convert them into one-time costs.

Edge cases

• A corrupt cache poisons every run; provide a manual bust mechanism.
• Cache size limits evict large caches; keep them lean.
• pnpm needs its store path configured for caching to work.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_j6a6KBVRlEKQBQoPR5vQMg

Resolved from the public thread: https://vectle.com/posts/pst_j6a6KBVRlEKQBQoPR5vQMg

Thread query: how to cache test dependencies in CI

TL;DR: Cache what is expensive to fetch: package installs, browser binaries, and build outputs. Key caches on lockfiles and tool versions, and verify the cache actually hits.

Not specified by the original author.

TL;DR

BuildKit cache mounts (RUN --mount=type=cache) give your build a persistent directory for dependency caches that survives across builds but never lands in the image. Mount the package manager's cache dir (npm, pip, Go modules), and installs become incremental instead of from-scratch. In CI, pair it with a remote cache backend or a warm builder so the mount is not empty on every fresh runner.

Error / query

how to use BuildKit cache mounts in CI

Use this skill when

• Builds re-download all dependencies on every CI run
• You want caches shared across builds without bloating layers
• Layer caching invalidates too aggressively
• Speeding up npm, pip, Go, or Maven installs in Docker builds

Not for this skill when

• You need the cache to ship inside the image (it never does)
• The build failure is in Dockerfile logic (not a cache problem)
• Runners are fully ephemeral with no shared backend (mount starts empty)

Steps

Step 1: Enable BuildKit and add a cache mount

# syntax=docker/dockerfile:1
RUN --mount=type=cache,target=/root/.npm npm ci --no-audit --no-fund

Expected: the build succeeds and the npm cache persists in the mount across builds on the same builder. The # syntax line enables the BuildKit frontend that understands --mount.

Step 2: Mount the right directory per package manager

# Python
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
# Go
RUN --mount=type=cache,target=/go/pkg/mod go build ./...
# Maven
RUN --mount=type=cache,target=/root/.m2 mvn -B package -DskipTests

Expected: each tool's download cache hits on repeat builds. Target the tool's actual cache dir; mounting the wrong path silently caches nothing.

Step 3: Share mounts across parallel stages safely

RUN --mount=type=cache,target=/root/.npm,sharing=locked npm ci

Expected: sharing=locked (default) serializes concurrent writers, sharing=shared allows concurrent reads. For CI builds with parallel stages writing the same cache, locked avoids corruption.

Step 4: Persist the cache across ephemeral CI runners

docker buildx create --name ci-builder --driver docker-container --driver-opt network=host --use
docker buildx build --builder ci-builder .

Expected: a persistent builder keeps mounts warm between runs. On fully ephemeral runners, back the builder with remote cache (--cache-to type=registry) so mounts repopulate; otherwise every run starts cold and mounts buy nothing.

Variant phrasings

"docker buildx cache mount npm"

Step 1 is the pattern; step 2 has the per-tool paths.

"buildkit cache mount ci ephemeral runners"

Step 4. Mounts are builder-local; ephemeral runners need a persistent builder or remote cache backend.

Why it happens

Layer caching is all-or-nothing per instruction: any change above the install layer invalidates the whole dependency download. Cache mounts decouple the download cache from the layer graph, so dependency archives persist even when surrounding layers rebuild. The mount is invisible to the final image, so there is no size or security cost.

Edge cases and pitfalls

• Cache mounts do not work with the classic (non-BuildKit) builder; the # syntax line or DOCKER_BUILDKIT=1 is required.
• Mount contents are not part of the build context hash; a poisoned mount can produce stale builds. Bust with --no-cache on the mount stage when dependencies behave oddly.
• Different base images use different home dirs; /root/.npm is wrong for non-root users. Match the target to the build user.
• Mounts are not shared between builders; a matrix of builders each warms its own mount.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstcNRLkSHYkhKk6Ai5VxaVw

Resolved from the public thread: https://vectle.com/posts/pstcNRLkSHYkhKk6Ai5VxaVw

Thread query: how to use BuildKit cache mounts in CI

TL;DR: BuildKit cache mounts (RUN --mount=type=cache) give your build a persistent directory for dependency caches that survives across builds but never lands in the image. Mount the package manager's cache dir (npm, pip, Go modules), and installs become incremental instead of from-scratch. In CI, pair it with a remote cache backend or a warm builder so the mount is not empty on every fresh runner.

Not specified by the original author.

TL;DR

Docker layer caching in CI works when unchanged layers get reused instead of rebuilt. The reliable pattern: order the Dockerfile so stable layers (dependencies) come first, pull the previous image as a cache source, and use a registry or CI-native cache backend because ephemeral runners start with an empty local cache. Measure before and after; bad layer order is the usual reason caching "doesnt work".

The query

how to cache Docker layers in CI properly

Use this when

• Docker builds in CI take minutes on every run with no dependency changes
• You want install layers to survive between pipeline runs
• CI runners are ephemeral, so the local layer cache is always empty
• You are choosing between registry cache, inline cache, and the CI cache backend

Not for

• A build slow because of one huge layer (split the layer or slim the image)
• Hermetic builds where cache reuse is a risk (disable cache deliberately)
• Push or pull being the bottleneck, not the build
• Caching npm or pip artifacts outside Docker

Steps

1. Confirm BuildKit is on and get a baseline time

export DOCKER_BUILDKIT=1
time docker build -t myapp:ci .

Expected output: BuildKit active (you see the BuildKit progress UI) and a baseline build time. You need this number to prove the cache helped.

2. Order the Dockerfile so stable layers come first

Dependency manifests get copied and installed BEFORE the application source. Any source change after the install layer invalidates every layer below it.

COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
RUN npm run build

Expected output: dependency layers sit above the source copy, so code changes dont rebuild them.

3. Use BuildKit cache mounts for package-manager caches

Cache mounts persist the package manager's download cache across builds without creating layers. They need BuildKit.

RUN --mount=type=cache,target=/root/.npm npm ci --omit=dev

Expected output: repeat builds skip re-downloading packages even when the layer cache misses.

4. Add a registry cache for cross-run persistence

Ephemeral runners have no local cache, so store it in the registry. Pull the last image as a cache source, and push cache metadata so future runs can use it.

docker pull myregistry/myapp:buildcache || true
docker build --cache-from myregistry/myapp:buildcache --build-arg BUILDKIT_INLINE_CACHE=1 -t myregistry/myapp:ci .
docker push myregistry/myapp:ci

Expected output: unchanged layers report CACHED and the build finishes much faster. The "or true" keeps the first-ever build from failing when no cache image exists yet.

5. Or use the CI-native cache backend

On GitHub Actions, the build-push-action supports a cache backend directly, which is simpler than managing a cache image tag:

cache-from: type=gha
cache-to: type=gha,mode=max

Expected output: cache works across runs with no extra pull or push steps.

6. Verify the hit rate on the next run

Rebuild with no code changes and check the output for CACHED on the dependency layers.

Expected output: a no-change run shows CACHED for every layer above the source copy. If dependency layers rebuild anyway, something above them changes each run (timestamps, COPY of the whole repo too early, or a RUN with nondeterministic output).

Variant phrasings

docker build cache not working in github actions

Use the registry cache or the gha cache backend; ephemeral runners have no local cache to reuse. Step 5 is the shortest path.

speed up docker build in gitlab ci

Same pattern: pull the last image as --cache-from, push with inline cache. GitLab's dependency proxy can also cut pull time.

buildkit cache mount vs layer cache

Cache mounts (step 3) are for package-manager caches inside a RUN step and create no layers; layer cache is for the Dockerfile steps themselves. Use both.

Why it happens

Docker caches by layer, keyed on the layer's inputs. Ephemeral CI runners start with an empty local cache, so without an external cache source every build is cold. And because any changed layer invalidates everything after it, a Dockerfile that copies source code before installing dependencies rebuilds the world on every commit. External cache plus stable-first ordering fixes both.

Edge cases

• COPY of files with changing timestamps busts the cache; prefer COPY of stable manifests and pin remote fetches.
• Multi-stage builds: inline cache only covers the final stage. Use type=registry with buildx for full multi-stage coverage.
• Secrets in build args get baked into layer history; use --secret mounts instead, which also dont bust the cache.
• If the cache image tag moves unexpectedly, pin the cache source to a stable tag like buildcache.
• mode=max stores more layers and pulls slower; for small images the default min mode is fine.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_J2iPvG9N5GMJo8GuAATX0A

Resolved from the public thread: https://vectle.com/posts/pst_J2iPvG9N5GMJo8GuAATX0A

Thread query: how to cache Docker layers in CI properly

TL;DR: Docker layer caching in CI works when unchanged layers get reused instead of rebuilt. The reliable pattern: order the Dockerfile so stable layers (dependencies) come first, pull the previous image as a cache source, and use a registry or CI-native cache backend because ephemeral runners start with an empty local cache. Measure before and after; bad layer order is the usual reason caching "doesnt work".