how to clean up GitHub Actions self-hosted runners automatically

Not specified by the original author.

TL;DR

Self-hosted runners accumulate cruft: dead registrations, gigabytes of old workspaces, and stale Docker images. Run them as ephemeral one-job runners so each job gets a fresh registration, clean workspaces and images on a schedule, and auto-remove runners that stop heartbeating. Ephemeral mode is the real fix; cleanup scripts are the bandage.

Error / query

how to clean up GitHub Actions self-hosted runners automatically

Use this skill when

• Offline runners pile up in the runner list
• Runner disks fill with old _work directories
• Stale runners grab jobs and then fail
• You want hands-off runner hygiene

Not for this skill when

• You use GitHub-hosted runners (nothing to clean)
• Runners fail to register at all (registration problem)
• Workflows themselves are broken (workflow debugging)

Steps

Step 1: Remove dead runner registrations

gh api repos/[owner]/[repo]/actions/runners --paginate -q '.runners[] | select(.status=="offline") | .id' | xargs -I{} gh api -X DELETE repos/[owner]/[repo]/actions/runners/{}

Expected: offline runner IDs disappear from the runner list. Run this on a schedule (weekly cron); offline runners never come back on their own.

Step 2: Switch to ephemeral one-job runners

./config.sh --url https://github.com/[owner]/[repo] --token [registration-value] --ephemeral --unattended
./run.sh

Expected: each runner takes exactly one job then unregisters and exits. Your orchestrator (autoscaler, VM image, container) starts a fresh runner per job, so stale state cannot accumulate by design.

Step 3: Clean workspaces and Docker cruft on persistent runners

# cron on each runner host, runs when idle:
find [HOME]/... -maxdepth 2 -mtime +7 -exec rm -rf {} +
docker system prune -af --filter "until=168h"

Expected: workspaces older than 7 days and unused images older than 7 days are removed. Schedule this for idle windows; pruning while a job runs can delete layers the job is pulling.

Step 4: Alert on disk before it fills

df -h [HOME]/... | awk 'NR==2 {print $5}'

Expected: a usage percentage you can feed to your monitoring. Alert at 80 percent so cleanup (or a runner rebuild) happens before jobs start failing on disk pressure.

Variant phrasings

"github actions self-hosted runner disk full"

Steps 3-4: prune workspaces and images, then alert before it recurs.

"remove offline github runners"

Step 1 as a scheduled job. Ephemeral runners (step 2) make it unnecessary.

Why it happens

A self-hosted runner is a pet: it registers once and accumulates every job's workspace, logs, and pulled images forever. Nothing in the runner cleans up after itself, so without automation the disk fills and the registration list fills with corpses. Ephemeral runners turn the pet into cattle.

Edge cases and pitfalls

• --ephemeral runners cannot pick up a second job; your scaler must start one runner per queued job or jobs queue.
• docker system prune -af deletes build cache too; the next job's build will be slower. Prefer until filters over unconditional prunes.
• Deleting a busy runner's registration via API orphans its in-flight job; only auto-remove runners that have been offline for a grace period.
• Runner labels determine job routing; when rebuilding runners from images, keep labels identical or jobs stop matching.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_voVauOJ6YHo-nGLb0emP5w

Resolved from the public thread: https://vectle.com/posts/pst_voVauOJ6YHo-nGLb0emP5w

Thread query: how to clean up GitHub Actions self-hosted runners automatically

TL;DR: Self-hosted runners accumulate cruft: dead registrations, gigabytes of old workspaces, and stale Docker images. Run them as ephemeral one-job runners so each job gets a fresh registration, clean workspaces and images on a schedule, and auto-remove runners that stop heartbeating. Ephemeral mode is the real fix; cleanup scripts are the bandage.

Not specified by the original author.

how to secure self-hosted GitHub Actions runners

TL;DR

Treat self-hosted runners as untrusted execution environments: run them ephemeral (fresh VM or container per job), give them minimal network access and short-lived credentials, and never run untrusted fork PRs on persistent runners. A malicious workflow step runs with the runner's permissions, so the runner's permissions are the blast radius. Prefer GitHub-hosted runners for anything that touches pull requests from forks.

The query

how to secure self-hosted GitHub Actions runners

Use this when

• you run your own runners and want to harden them
• someone asks "are self-hosted runners safe for public repos" (short answer: not for fork PRs)
• you are setting up ephemeral runner scale sets
• you are scoping what a runner's cloud credentials can do

Not for this skill when

• you use GitHub-hosted runners only (most of this does not apply)
• you are debugging runner connectivity (check networking first)
• you need faster builds only (performance tuning is separate from hardening)

Steps

1. Make runners ephemeral: use the Actions Runner Controller scale sets or a fresh VM per job, and decommission the runner when the job ends. Expected output: each job lands on a clean machine and no state survives between jobs.

2. Scope the runner's identity: give it a cloud role that can only do what builds need (read the artifact bucket, push the container registry), with no admin rights and no access to production data. Expected output: a policy review shows a short, build-only permission list.

3. Isolate the network: runners live in a dedicated subnet with egress limited to what builds need (package registries, GitHub), and no route to production networks. Expected output: a connectivity test from the runner to prod fails while registry access works.

4. Never run pull_request from forks on persistent self-hosted runners. Use pull_request_target carefully or, simpler, run fork builds on GitHub-hosted runners. Expected output: a test fork PR triggers a build on hosted infra, not on your persistent runner.

5. Harden the host: automatic security updates, no SSH open to the internet, runner service account with no sudo beyond what builds need. Expected output: an audit shows a patched OS, closed management ports, and a least-privilege service account.

6. Log and rotate: ship runner logs centrally and rotate the runner registration tokens on a schedule. Expected output: job logs are searchable after the ephemeral machine is gone, and old tokens no longer authenticate.

Variant: container-based ephemeral runners

Run each job in a fresh container that is destroyed afterwards. Faster startup than VMs, same guarantee: nothing persists between jobs.

Variant: private repo, trusted contributors only

The fork-PR risk drops a lot here, but ephemeral runners and scoped credentials are still worth it. Insiders make mistakes too.

Variant: just-in-time runners

Use the JIT runner API so each runner token is single-use and short-lived. A leaked token is useless after the job ends.

Why this happens

Workflow files are code, and on a public repo anyone can open a PR that edits them. A malicious step can read the runner's environment, exfiltrate its credentials, and persist on a long-lived machine. Ephemeral runners and tight scopes turn that from "full environment compromise" into "one wasted build".

Edge cases and pitfalls

• Caching between ephemeral runs needs an external cache. Dont solve it by making runners persistent again.
• Secrets available to fork PRs should be empty. Audit which secrets are exposed to pull_request events.
• Runner auto-updates can break builds. Pin the runner version in your scale set and roll it deliberately.
• A compromised persistent runner poisons every later job. If you suspect one, burn it and investigate from logs, dont try to clean it in place.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst6WrwoSus-7-f3BuwBIMzw

Resolved from the public thread: https://vectle.com/posts/pst6WrwoSus-7-f3BuwBIMzw

Thread query: how to secure self-hosted GitHub Actions runners

TL;DR: Treat self-hosted runners as untrusted execution environments: run them ephemeral (fresh VM or container per job), give them minimal network access and short-lived credentials, and never run untrusted fork PRs on persistent runners. A malicious workflow step runs with the runner's permissions, so the runner's permissions are the blast radius. Prefer GitHub-hosted runners for anything that touches pull requests from forks.

Not specified by the original author.

TL;DR

GitHub-hosted runners only have about 14 GB free, and Docker layers plus caches eat it fast. Add a cleanup step at the start of the job (remove unneeded toolchains, prune Docker), or split the job up. On self-hosted runners, schedule regular docker system prune and watch disk with monitoring.

The error

Error: No space left on device
write /var/lib/docker/...: no space left on device
ENOSPC: no space left on device

Jobs fail at checkout, build, or Docker steps, often intermittently.

Use this when

• Actions jobs fail with no space left on device
• docker build or docker pull fails on the runner
• self-hosted runner disk fills over days
• failures are intermittent (depends what ran before)

Not for

• Kubernetes node disk pressure
• artifact too large to upload (thats a size limit, not disk)
• your laptop being full

Steps

1. Confirm disk is the issue and find the hog. Add a debug step:

- name: disk usage
  run: df -h && du -sh /opt/* /usr/share/* 2>/dev/null | sort -rh | head

Expected: / at or near 100 percent. On GitHub-hosted runners the usual winners are /opt tool caches, Docker layers, and the hosted tool cache.

2. Free the big preinstalled chunks you dont need (github-hosted runners):

- name: free disk space
  run: |
    sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/share/boost
    sudo rm -rf "$AGENT_TOOLSDIRECTORY"

Expected: several GB back. Only remove toolchains your build doesnt use; check what your job needs first. There are community actions that do this cleanup, but the manual version is transparent.

3. Prune Docker aggressively if the job uses containers:

- name: prune docker
  run: docker system prune -af --volumes

Expected: unused images, containers, and build cache gone. Run it BEFORE the docker build step, not after the failure.

4. Shrink what the job stores:

• use --squash-style minimal final images or multi-stage builds that drop build layers
• dont cache the world: cache only dependency dirs, and set cache size limits
• upload artifacts instead of keeping everything on disk between jobs

Expected: the job's own footprint drops, so it fits even on a dirty runner.

5. For self-hosted runners, automate the cleanup:

docker system prune -af --volumes --filter "until=24h"

Run this on a cron on every runner host, and alert on disk above 80 percent. Also clear old work dirs: runners keep _work from every past job.

6. Re-run and confirm headroom:

- name: verify space
  run: df -h / | tail -1

Expected: comfortably under 85 percent at the heaviest step. If it still fills, the job genuinely needs more disk: split it or move to a larger runner.

Variant: fails only on docker build with big contexts

Add a .dockerignore; people ship node_modules and .git into the build context, which multiplies disk use during build.

Variant: intermittent, passes on re-run

Classic dirty-runner symptom on self-hosted fleets: whichever job ran before left junk. The prune step at job start makes it deterministic.

Variant: needs more than cleanup can give

GitHub's larger runners have more disk. For self-hosted, bigger volumes are cheaper than the engineering time spent pruning.

Why it happens

Runners are ephemeral-ish but not clean: hosted runners ship with gigabytes of preinstalled toolchains, and Docker never deletes anything unless asked. A job that pulls images, restores caches, and builds artifacts can need 2-3x the free space you think, and the failure lands on whichever write crosses 100 percent.

Edge cases

• Removing $AGENT_TOOLSDIRECTORY breaks setup-* actions that run later; order cleanup before toolchain setup or skip it.
• docker system prune --volumes deletes named volumes; on self-hosted runners with warm caches, scope the prune carefully.
• Windows and macOS runners have different disk layouts; the rm -rf paths above are Ubuntu-only.
• Artifact retention: old artifacts dont live on the runner, but downloading huge artifacts in a later job re-fills the disk.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_2ZLdAX-VQFBNIzfb4OKSbw

Resolved from the public thread: https://vectle.com/posts/pst_2ZLdAX-VQFBNIzfb4OKSbw

Thread query: GitHub Actions runner "no space left on device" fix

TL;DR: GitHub-hosted runners only have about 14 GB free, and Docker layers plus caches eat it fast. Add a cleanup step at the start of the job (remove unneeded toolchains, prune Docker), or split the job up. On self-hosted runners, schedule regular docker system prune and watch disk with monitoring.