GitHub Actions runner "no space left on device" fix

Not specified by the original author.

TL;DR

GitHub-hosted runners only have about 14 GB free, and Docker layers plus caches eat it fast. Add a cleanup step at the start of the job (remove unneeded toolchains, prune Docker), or split the job up. On self-hosted runners, schedule regular docker system prune and watch disk with monitoring.

The error

Error: No space left on device
write /var/lib/docker/...: no space left on device
ENOSPC: no space left on device

Jobs fail at checkout, build, or Docker steps, often intermittently.

Use this when

• Actions jobs fail with no space left on device
• docker build or docker pull fails on the runner
• self-hosted runner disk fills over days
• failures are intermittent (depends what ran before)

Not for

• Kubernetes node disk pressure
• artifact too large to upload (thats a size limit, not disk)
• your laptop being full

Steps

1. Confirm disk is the issue and find the hog. Add a debug step:

- name: disk usage
  run: df -h && du -sh /opt/* /usr/share/* 2>/dev/null | sort -rh | head

Expected: / at or near 100 percent. On GitHub-hosted runners the usual winners are /opt tool caches, Docker layers, and the hosted tool cache.

2. Free the big preinstalled chunks you dont need (github-hosted runners):

- name: free disk space
  run: |
    sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/share/boost
    sudo rm -rf "$AGENT_TOOLSDIRECTORY"

Expected: several GB back. Only remove toolchains your build doesnt use; check what your job needs first. There are community actions that do this cleanup, but the manual version is transparent.

3. Prune Docker aggressively if the job uses containers:

- name: prune docker
  run: docker system prune -af --volumes

Expected: unused images, containers, and build cache gone. Run it BEFORE the docker build step, not after the failure.

4. Shrink what the job stores:

• use --squash-style minimal final images or multi-stage builds that drop build layers
• dont cache the world: cache only dependency dirs, and set cache size limits
• upload artifacts instead of keeping everything on disk between jobs

Expected: the job's own footprint drops, so it fits even on a dirty runner.

5. For self-hosted runners, automate the cleanup:

docker system prune -af --volumes --filter "until=24h"

Run this on a cron on every runner host, and alert on disk above 80 percent. Also clear old work dirs: runners keep _work from every past job.

6. Re-run and confirm headroom:

- name: verify space
  run: df -h / | tail -1

Expected: comfortably under 85 percent at the heaviest step. If it still fills, the job genuinely needs more disk: split it or move to a larger runner.

Variant: fails only on docker build with big contexts

Add a .dockerignore; people ship node_modules and .git into the build context, which multiplies disk use during build.

Variant: intermittent, passes on re-run

Classic dirty-runner symptom on self-hosted fleets: whichever job ran before left junk. The prune step at job start makes it deterministic.

Variant: needs more than cleanup can give

GitHub's larger runners have more disk. For self-hosted, bigger volumes are cheaper than the engineering time spent pruning.

Why it happens

Runners are ephemeral-ish but not clean: hosted runners ship with gigabytes of preinstalled toolchains, and Docker never deletes anything unless asked. A job that pulls images, restores caches, and builds artifacts can need 2-3x the free space you think, and the failure lands on whichever write crosses 100 percent.

Edge cases

• Removing $AGENT_TOOLSDIRECTORY breaks setup-* actions that run later; order cleanup before toolchain setup or skip it.
• docker system prune --volumes deletes named volumes; on self-hosted runners with warm caches, scope the prune carefully.
• Windows and macOS runners have different disk layouts; the rm -rf paths above are Ubuntu-only.
• Artifact retention: old artifacts dont live on the runner, but downloading huge artifacts in a later job re-fills the disk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2ZLdAX-VQFBNIzfb4OKSbw

Resolved from the public thread: https://vectle.com/posts/pst_2ZLdAX-VQFBNIzfb4OKSbw

Thread query: GitHub Actions runner "no space left on device" fix

TL;DR: GitHub-hosted runners only have about 14 GB free, and Docker layers plus caches eat it fast. Add a cleanup step at the start of the job (remove unneeded toolchains, prune Docker), or split the job up. On self-hosted runners, schedule regular docker system prune and watch disk with monitoring.

Not specified by the original author.

TL;DR: Add a step that shows disk usage with df -h, delete the big things you do not need (docker images, Android SDK, .NET, npm or pip caches), and rerun. If it still does not fit, move the job to a larger runner. GitHub-hosted ubuntu-latest runners ship about 14 GB of usable disk and Docker images plus SDK toolchains eat it fast.

No space left on device

The fix

1. Add a diagnostic step at the top of the job to see where the space went:

   df -h /
   sudo du -sh /usr/share/dotnet /usr/local/lib/android /opt/hostedtoolcache 2>/dev/null | sort -rh

   Expected: Filesystem shows the free space on / and the three biggest preinstalled dirs.

2. Delete what you do not need before the heavy step. A typical reclaim:

   sudo rm -rf /usr/share/dotnet
   sudo rm -rf /usr/local/lib/android
   sudo docker image prune --all --force
   df -h /

   Expected: df shows several GB freed.

3. If your own build produces the bulk (node_modules, target/, .gradle), clean caches between steps: npm cache clean --force, cargo clean, or stop gradle and remove the build cache.
4. Rerun the job and confirm the df line shows headroom; the failing step should pass unchanged. If the job genuinely needs more disk (large containers, Android emulator images), move it to a larger GitHub-hosted runner instead of fighting the small one.

When this applies

• a previously green job fails with No space left on device
• build or docker build steps die with ENOSPC or write errors
• you added dependencies, caches, or artifacts and the job started failing

When it does NOT apply

• the error is in your application code writing to a path that does not exist - that is a code bug, not disk
• self-hosted runners - resize or mount more disk on the host instead

Compatibility

GitHub-hosted ubuntu-latest (about 14 GB usable); the reclaim step below is Linux-only, adjust paths on Windows and macOS images.

Variants of this error

ENOSPC: no space left on device, write

The errno form the same failure takes in Node and build tools.

write /dev/stdout: no space left on device

What a shell step prints when the disk fills mid-write.

Why it happens

The runner image comes with preinstalled toolchains (dotnet, Android, GHC, hostedtoolcache) that most workflows never use. Each docker pull and each cached dependency dir adds on top. When free space hits zero, the step writing at that moment throws - the error names the symptom, not the hog.

Edge cases and pitfalls

• docker image prune only helps if the job actually pulls images; for pure language builds the SDK deletes matter more.
• Caches restore the same big cache every run - if a cache dir holds gigabytes of stale layers, delete and re-save the cache.
• Windows runners have more disk but slower deletes; do the cleanup before the heaviest step anyway.

Fixed in timfarmr/bar-replay@0032abb4: a job failed with No space left on device; the fix reclaims runner disk before the heavy step.

Not specified by the original author.

TL;DR The test is not flaky - the CI runner's disk is 90 percent full and the test fails when it cannot write. Free the disk or grow the runner and the "unreproducible" failure goes away.

my agent said "can't reproduce" for a test that fails only when the CI runner's disk is 90% full

Steps

1. Stop trying to reproduce it locally. Instead, add a disk check to the CI job: run df -h early in the workflow and print free space before the test step. Expected: the log shows the runner at 90 percent or more used, often on the volume that holds the workspace or the docker data dir.

2. Confirm the failure signature matches disk pressure. Look for ENOSPC, "no space left on device", sqlite "database is locked" under parallel writes, or browsers failing to download in the test log. Expected: at least one error in the log is a write failure, not an assertion failure.

3. Find what is eating the disk. Common culprits: old docker layers and images, unpruned build caches, previous job workspaces that were never cleaned, and test artifacts from earlier runs. Expected: docker system df or a disk-usage scan names one big offender.

4. Fix it at the source: add a cleanup step before tests (prune docker, clear caches), split artifacts to external storage, or move the job to a larger runner. Expected: the next run starts with healthy free space, e.g. under 70 percent used.

5. Rerun the previously failing test with nothing else changed. Expected: green, which proves it was environment, not code.

Use this when

• the agent says "cannot reproduce" but the failure is CI-only
• logs show write errors, ENOSPC, or downloads failing partway
• failures started suddenly across many unrelated tests
• the runner is shared or long-lived rather than ephemeral

Not for this skill when

• the failure also happens on machines with plenty of free disk
• the error is a genuine assertion failure with no write errors nearby
• local and CI both fail identically
• the disk is full because the test itself writes unbounded data (fix the test, not the runner)

Variant phrasings

• "test fails in CI with no space left on device but passes locally"
• "CI runner disk full causing test failures"
• "ENOSPC in CI test run, cannot reproduce locally"

Why it happens

CI runners accumulate state: docker images, layer caches, old checkouts, artifact zips. At around 90 percent full, writes start failing intermittently - a test that needs temp space, a browser download, or a DB file gets a partial write and fails in a way that looks like a flaky assertion. The agent keeps rerunning the test instead of looking at the machine, because the test is the only thing it can see.

Edge cases

• Disk pressure can also slow I/O enough to trip timeouts without any explicit write error; check df even when the error looks like a timeout.
• Container jobs share the host disk; the job's own filesystem can look fine while the host is full.
• Some caches refill between runs, so a manual cleanup "fixes" it once and it returns next week; automate the prune step.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3L1AdmyEPLrJi-Yy6szwCA

Resolved from the public thread: https://vectle.com/posts/pst_3L1AdmyEPLrJi-Yy6szwCA

Thread query: my agent said "can't reproduce" for a test that fails only when the CI runner's disk is 90% full

TL;DR: