## TL;DR
When a Dockerfile fails at step 12 of 20, do not rebuild from scratch to debug it: build up to the failing layer with `--target`, or drop into the last good layer and run the failing command by hand. Buildx lets you stop at any stage and inspect the filesystem exactly as the build saw it, which turns a blind rebuild loop into a direct investigation.

## Error / query
```text
how to debug Dockerfile layer-by-layer with buildx
```

## Use this skill when
- A build fails at a specific Dockerfile instruction
- You need to inspect files or env inside an intermediate layer
- A layer's output differs from what you expect
- Bisecting which instruction introduced a breakage

## Not for this skill when
- The build is slow but succeeds (cache tuning)
- The failure is at push time (registry issue)
- The container misbehaves at runtime (runtime debugging)

## Steps

### Step 1: Build only up to the failing stage
```bash
docker buildx build --target [stage-before-failure] -t debug:[tag] .
```
Expected: the build stops at the named stage and tags the intermediate image. You now have the exact filesystem state the failing instruction started from.

### Step 2: Run the failing instruction interactively
```bash
docker run --rm -it debug:[tag] sh
# then inside: run the failing RUN command by hand
```
Expected: you reproduce the failure (or see it succeed) with full terminal output, env, and the ability to inspect files. Interactive runs show what build logs truncate.

### Step 3: Inspect layer contents without running anything
```bash
docker buildx build --target [stage] --output type=docker -t debug:[tag] . && docker run --rm debug:[tag] ls -la [suspect-path]
```
Expected: you can check whether files, permissions, or env are what the next instruction expects. Most layer bugs are wrong paths or missing files from a previous COPY.

### Step 4: Bisect with targeted rebuilds
```bash
docker buildx build --progress=plain --no-cache-filter=[stage] -t debug:[tag] .
```
Expected: only the suspect stage rebuilds without cache while earlier stages reuse cache, so each bisect iteration is fast. Move the `--no-cache-filter` down the Dockerfile until the failure appears.

## Variant phrasings

### "debug docker build intermediate layer"
Steps 1-2: target the stage, then run it interactively.

### "dockerfile fails at step"
Build to the previous stage (step 1) and execute the failing instruction by hand (step 2).

## Why it happens
Docker builds are pipelines: each instruction transforms the previous layer's filesystem, and failures report only the tail of the output. Without stopping mid-pipeline you cannot see the state the failing command actually ran against, so debugging becomes guess-and-rebuild. Targets and intermediate tags expose that state.

## Edge cases and pitfalls
- Build secrets and SSH mounts are not available in `docker run` on the debug image; reproduce secret-dependent steps with the same mounts or dummy values.
- Multi-platform builds debug per-platform; add `--platform` to match the failing architecture.
- `--target` on the final stage is a no-op; name an earlier stage explicitly.
- Debug tags pile up; remove them when done or they consume disk like any image.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_cfOOZEFFhAk4FvnfDicO4A