how to debug Dockerfile layer-by-layer with buildx

TL;DR

When a Dockerfile fails at step 12 of 20, do not rebuild from scratch to debug it: build up to the failing layer with --target, or drop into the last good layer and run the failing command by hand. Buildx lets you stop at any stage and inspect the filesystem exactly as the build saw it, which turns a blind rebuild loop into a direct investigation.

Error / query

how to debug Dockerfile layer-by-layer with buildx

Use this skill when

• A build fails at a specific Dockerfile instruction
• You need to inspect files or env inside an intermediate layer
• A layer's output differs from what you expect
• Bisecting which instruction introduced a breakage

Not for this skill when

• The build is slow but succeeds (cache tuning)
• The failure is at push time (registry issue)
• The container misbehaves at runtime (runtime debugging)

Steps

Step 1: Build only up to the failing stage

docker buildx build --target [stage-before-failure] -t debug:[tag] .

Expected: the build stops at the named stage and tags the intermediate image. You now have the exact filesystem state the failing instruction started from.

Step 2: Run the failing instruction interactively

docker run --rm -it debug:[tag] sh
# then inside: run the failing RUN command by hand

Expected: you reproduce the failure (or see it succeed) with full terminal output, env, and the ability to inspect files. Interactive runs show what build logs truncate.

Step 3: Inspect layer contents without running anything

docker buildx build --target [stage] --output type=docker -t debug:[tag] . && docker run --rm debug:[tag] ls -la [suspect-path]

Expected: you can check whether files, permissions, or env are what the next instruction expects. Most layer bugs are wrong paths or missing files from a previous COPY.

Step 4: Bisect with targeted rebuilds

docker buildx build --progress=plain --no-cache-filter=[stage] -t debug:[tag] .

Expected: only the suspect stage rebuilds without cache while earlier stages reuse cache, so each bisect iteration is fast. Move the --no-cache-filter down the Dockerfile until the failure appears.

Variant phrasings

"debug docker build intermediate layer"

Steps 1-2: target the stage, then run it interactively.

"dockerfile fails at step"

Build to the previous stage (step 1) and execute the failing instruction by hand (step 2).

Why it happens

Docker builds are pipelines: each instruction transforms the previous layer's filesystem, and failures report only the tail of the output. Without stopping mid-pipeline you cannot see the state the failing command actually ran against, so debugging becomes guess-and-rebuild. Targets and intermediate tags expose that state.

Edge cases and pitfalls

• Build secrets and SSH mounts are not available in docker run on the debug image; reproduce secret-dependent steps with the same mounts or dummy values.
• Multi-platform builds debug per-platform; add --platform to match the failing architecture.
• --target on the final stage is a no-op; name an earlier stage explicitly.
• Debug tags pile up; remove them when done or they consume disk like any image.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cfOOZEFFhAk4FvnfDicO4A