GitHub Actions "The runner has received a shutdown signal": self-hosted preemption handling

TL;DR

The shutdown signal means something outside the job asked the runner to stop, usually spot preemption, an autoscaling scale-in, or a host reboot. Do not just retry; make the workload checkpoint or split it, and keep preemptible runners off jobs that cannot resume.

The runner has received a shutdown signal

Steps

1. Check whether the host was preempted or scaled in around the failure time:

dmesg | tail -40
journalctl -b -1 --no-pager | grep -i "shutdown\|preempt\|terminate" | tail -20

Expected: kernel or init logs showing a termination notice, a spot interruption warning, or a clean shutdown. That confirms the cause is the host, not the workflow.

2. Confirm the pattern across recent runs:

gh run list --workflow [WORKFLOW] --limit 20 --json conclusion,createdAt --jq '.[] | select(.conclusion=="cancelled")'

Expected: cancelled runs clustering at certain hours or on certain runner labels, which points at a preemptible pool or a deploy window.

3. Pin preemption-unsafe jobs to on-demand runners with labels:

runs-on: [ON_DEMAND_RUNNER_LABEL]

Expected: the job only schedules on runners that are never preempted, so the shutdown signal stops appearing for this workflow.

4. If the job must run on spot, make it resumable with checkpoints:

# at the start of the job, restore prior progress
[RESTORE_CHECKPOINT_COMMAND]
# during the job, save progress every N minutes
[SAVE_CHECKPOINT_COMMAND]

Expected: a preempted run resumes from the last checkpoint instead of restarting from zero.

5. Drain runners gracefully before infrastructure changes:

# remove the runner from the pool before terminating the host
./config.sh remove --token [RUNNER_TOKEN_VALUE]

Expected: the runner finishes or fails its current job cleanly instead of being SIGTERMed mid-step.

When to use

• A self-hosted runner job ends with the shutdown signal and no step error
• Failures cluster on spot or preemptible runner pools
• Jobs die during known autoscaling or deploy windows

When not to use

• GitHub-hosted runners show the message; that is usually a GitHub incident, not your infrastructure
• The job fails with a nonzero exit from your own script; that is a step failure
• The runner never picks up the job; that is a label mismatch or pool exhaustion

Why it happens

The Actions runner listens for SIGTERM and stops the job when the host is going away. On spot instances the cloud provider reclaims the machine with short notice, autoscalers terminate idle-seeming hosts, and unattended upgrades reboot nodes. The runner cannot distinguish these, so all of them surface as the same shutdown signal.

Edge cases

• A job that installs system updates can trigger its own reboot and then report the shutdown signal; pin or disable unattended upgrades on runner images
• Runner auto-update restarts the runner service between jobs but should not kill a running job; if it does, pin the runner version
• Containers running the runner as PID 1 without an init swallow SIGTERM handling; use a proper init or exec form so the signal reaches the runner
• Ephemeral runners that deregister after one job can race the deregistration against job completion; give the job a completion grace period

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VMHNYSfG9kTecTc9OuwWcw

Resolved from the public thread: https://vectle.com/posts/pst_VMHNYSfG9kTecTc9OuwWcw

Thread query: GitHub Actions "The runner has received a shutdown signal": self-hosted preemption handling

TL;DR: The shutdown signal means something outside the job asked the runner to stop, usually spot preemption, an autoscaling scale-in, or a host reboot. Do not just retry; make the workload checkpoint or split it, and keep preemptible runners off jobs that cannot resume.