how to keep secrets out of container image layers

how to keep secrets out of container image layers

TL;DR

Every RUN, COPY, ENV, and ARG in your Dockerfile can leave a secret baked into an image layer, visible to anyone who pulls the image. For build-time credentials use BuildKit secret mounts, which never persist to a layer. Never ENV or ARG a secret, keep credential files out of the build context with .dockerignore, and scan the built image for secrets before pushing. If a secret already shipped in a pushed image, rotate it, there is no unshipping it.

how to keep secrets out of container image layers

Use this when

• A scanner flags a secret embedded in an image
• Your Dockerfile needs a private registry login, API key, or signing key at build time
• You are writing build policy for a team
• docker history shows something it should not

Not for this skill when

• You need to inject secrets into running containers (use your orchestrator's secret mechanism)
• The secret is in git history rather than an image (different cleanup problem)
• You are choosing a secret manager (different skill)

Steps

1. Check whether the secret is already baked in

docker history --no-trunc myapp:latest | grep -iE "secret|password" | head

Expected: ideally nothing. Any hit means a layer references a secret, inspect the full history output to see which instruction leaked it. Also check ENV values, which history shows in plain text.

2. Use BuildKit secret mounts for build-time credentials

In the Dockerfile, mount the secret only for the RUN step that needs it:

RUN --mount=type=secret,id=[secret-id] [command that reads the secret from /run/secrets/[secret-id]]

DOCKER_BUILDKIT=1 docker build --secret id=[secret-id],src=[path-to-secret-file] -t myapp:latest .

Expected: the build succeeds, and re-running the history check from step 1 shows no trace of the secret. The mounted secret exists only during that RUN step and never lands in a layer.

3. Keep secrets out of the final stage in multi-stage builds

Do the credential-needing work (private dependency fetch, signing) in the builder stage, and COPY only the built artifacts into the final stage. Then confirm the final stage is clean:

docker build --target final -t myapp:final . && docker history --no-trunc myapp:final

Expected: final-stage layers show only artifact COPY instructions, no secret-related RUN steps and no credential ENV values.

4. Keep credential files out of the build context

Create a .dockerignore file containing these patterns:

.env
*.pem
.aws/
credentials.json

cat .dockerignore

Expected: the patterns listed above. Files matching .dockerignore never reach the daemon, so a careless COPY . cannot sweep them into a layer. Build once and confirm the files are absent from the image.

5. Scan the image before pushing

trivy image --scanners secret myapp:latest

Expected: no findings. If it lists embedded secrets, go back to step 2 or 3, fix the Dockerfile, rebuild, and rotate every secret that was found, since the old image may already be in a registry.

Variant: the secret is only needed at runtime

Do not bake it at all. Pass it via your orchestrator's secret mechanism (Kubernetes secrets, ECS task secrets, env from a secret manager) so it never exists in the image. Build-time tricks are for build-time needs only.

Variant: a secret already shipped in a pushed image

Rotate the secret immediately, then fix the Dockerfile. Deleting the image from the registry does not help, registries are cached and layers are content-addressed. Assume the secret is compromised and act accordingly.

Variant: ARG vs ENV for secrets

Neither is safe. ARG values appear in docker history, ENV values appear in history and in the running container's environment. Both end up in layers. Use secret mounts.

Why this happens

Image layers are permanent and content-addressed, and anyone who can pull the image can inspect every layer. Developers reach for ENV and ARG because they are convenient, not realizing the Dockerfile is a permanent record. Secret mounts exist precisely because the convenient options are all leaky.

Edge cases and pitfalls

• Secret mounts need BuildKit. The classic builder does not support --secret and will fail the build, which is actually the safe failure mode.
• Squashing layers does not reliably remove secrets from pushed images. Do not rely on it.
• .dockerignore does not apply to --secret mounts, which is fine because mounted secrets were never in the build context.
• Multi-line secrets and special characters in secret files can break naive RUN parsing. Quote carefully.
• CI systems that echo build args to logs can leak secrets a second way. Mask them in the pipeline too.
• Base images can contain secrets from their own build. Scan the base image, not just your layers.

Tool notes: BuildKit secret mounts need Docker 18.09+. trivy secret scanning is built into recent trivy releases.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_QGRbhb82pKet8d2TLLn0LQ