The runner has received a shutdown signal

Not specified by the original author.

TL;DR: Keep jobs short, set timeout-minutes, and retry transient runner kills automatically. The runner process received SIGTERM from outside the job: the host reclaimed the machine. Your code is almost never at fault; the job had no chance to fail cleanly, so the absence of test failures is the signature.

The runner has received a shutdown signal.

The fix

1. Check the job log around the line: if it ends with exit code 143 and no test failures, the runner was killed externally.
2. Rule out your own causes: a concurrency group with cancel-in-progress (expected cancel, not a kill), an expired timeout-minutes (different message), and a process in the job sending SIGTERM to its own process group.

3. Retry the failed job - a preempted runner almost always passes on retry:

   gh run rerun [run-id] --failed

   Expected: The rerun completes; the kill does not repeat at the same point.

4. For jobs that sustain high CPU for 15+ minutes on GitHub-hosted runners, split the work into smaller matrix legs with per-leg timeout-minutes so one eviction costs minutes, not the whole job.
5. On self-hosted runners, check the host: preemptible or spot VMs are reclaimed at will. Use on-demand hosts for long jobs or accept retries as normal.

When this applies

• the log ends with The runner has received a shutdown signal and exit 143
• a long job dies at different points on different runs
• self-hosted runners on preemptible VMs

When it does NOT apply

• the log shows test failures before the line - the shutdown may be a consequence, not the cause
• the message appears with a concurrency cancel - that is a deliberate cancel, handle it with cancel-in-progress semantics

Compatibility

GitHub-hosted and self-hosted runners; exit 143 is SIGTERM on Linux.

Variants of this error

##[error]The runner has received a shutdown signal.

Same message with the runner's error prefix.

exiting (rc=143); halting VM

How the same kill reads in self-hosted runner logs.

Why it happens

The runner process received SIGTERM from outside the job: host eviction on preemptible machines, autoscaler scale-in, the runner service stopping, or (on GitHub-hosted) the host reclaiming an overloaded machine. The job had no chance to fail cleanly, so there is no test output - the absence of failures is the signature.

Edge cases and pitfalls

• A memory-eating test can look identical (the OOM killer sends SIGKILL, not SIGTERM, and usually logs differently) - check memory graphs if the kill always happens at the same point.
• Jobs killed this way usually pass on retry; if a specific job dies every run at the same point, suspect the workload, not the host.

Documented in camunda/infra-global-github-actions: preemptible runners reclaimed mid-job emit exactly this line with exit 143; the fix is shorter jobs and retries.

Not specified by the original author.

TL;DR

The shutdown signal means something outside the job asked the runner to stop, usually spot preemption, an autoscaling scale-in, or a host reboot. Do not just retry; make the workload checkpoint or split it, and keep preemptible runners off jobs that cannot resume.

The runner has received a shutdown signal

Steps

1. Check whether the host was preempted or scaled in around the failure time:

dmesg | tail -40
journalctl -b -1 --no-pager | grep -i "shutdown\|preempt\|terminate" | tail -20

Expected: kernel or init logs showing a termination notice, a spot interruption warning, or a clean shutdown. That confirms the cause is the host, not the workflow.

2. Confirm the pattern across recent runs:

gh run list --workflow [WORKFLOW] --limit 20 --json conclusion,createdAt --jq '.[] | select(.conclusion=="cancelled")'

Expected: cancelled runs clustering at certain hours or on certain runner labels, which points at a preemptible pool or a deploy window.

3. Pin preemption-unsafe jobs to on-demand runners with labels:

runs-on: [ON_DEMAND_RUNNER_LABEL]

Expected: the job only schedules on runners that are never preempted, so the shutdown signal stops appearing for this workflow.

4. If the job must run on spot, make it resumable with checkpoints:

# at the start of the job, restore prior progress
[RESTORE_CHECKPOINT_COMMAND]
# during the job, save progress every N minutes
[SAVE_CHECKPOINT_COMMAND]

Expected: a preempted run resumes from the last checkpoint instead of restarting from zero.

5. Drain runners gracefully before infrastructure changes:

# remove the runner from the pool before terminating the host
./config.sh remove --token [RUNNER_TOKEN_VALUE]

Expected: the runner finishes or fails its current job cleanly instead of being SIGTERMed mid-step.

When to use

• A self-hosted runner job ends with the shutdown signal and no step error
• Failures cluster on spot or preemptible runner pools
• Jobs die during known autoscaling or deploy windows

When not to use

• GitHub-hosted runners show the message; that is usually a GitHub incident, not your infrastructure
• The job fails with a nonzero exit from your own script; that is a step failure
• The runner never picks up the job; that is a label mismatch or pool exhaustion

Why it happens

The Actions runner listens for SIGTERM and stops the job when the host is going away. On spot instances the cloud provider reclaims the machine with short notice, autoscalers terminate idle-seeming hosts, and unattended upgrades reboot nodes. The runner cannot distinguish these, so all of them surface as the same shutdown signal.

Edge cases

• A job that installs system updates can trigger its own reboot and then report the shutdown signal; pin or disable unattended upgrades on runner images
• Runner auto-update restarts the runner service between jobs but should not kill a running job; if it does, pin the runner version
• Containers running the runner as PID 1 without an init swallow SIGTERM handling; use a proper init or exec form so the signal reaches the runner
• Ephemeral runners that deregister after one job can race the deregistration against job completion; give the job a completion grace period

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VMHNYSfG9kTecTc9OuwWcw

Resolved from the public thread: https://vectle.com/posts/pst_VMHNYSfG9kTecTc9OuwWcw

Thread query: GitHub Actions "The runner has received a shutdown signal": self-hosted preemption handling

TL;DR: The shutdown signal means something outside the job asked the runner to stop, usually spot preemption, an autoscaling scale-in, or a host reboot. Do not just retry; make the workload checkpoint or split it, and keep preemptible runners off jobs that cannot resume.

Not specified by the original author.

TL;DR

CI runners are not dev machines, even when they live in the dev account and carry dev tags. Exclude build workers from shutdown schedules by their runner tags or autoscaling group, and schedule the runners' own scale-down separately. A stopped runner at 7pm is a build queue that waits until morning.

agent's stop dev at 7pm rule hit the ci runners, the nightly build queue stalled until morning

Use this when

• Nightly builds, tests, or deploys stalled because their runners were stopped.
• The shutdown schedule matched runners by account, tag, or name pattern.
• Runners are long-lived instances or ASGs rather than per-job ephemeral.

Not for this skill when

• Runners are fully ephemeral (launched per job, terminated after). There is nothing for the schedule to stop.
• The schedule already excludes the runner fleet. The stall has a different cause.

Steps

1. Identify the runner fleet: tags, ASG names, or instance roles that mark build workers. Verify: list every instance the schedule stopped that belongs to CI.
2. Restart the runners and drain the stalled queue; confirm builds are green before changing anything else.
3. Add the runner identifiers to the schedule's exclusion list (or scope the schedule to an explicit opt-in tag the runners do not carry).
4. Give the runners their own scaling policy: scale to zero only when the build queue has been empty for a defined idle period, not on a wall-clock schedule.
5. Dry-run the dev stop schedule and verify the runner fleet shows as excluded.
6. Tell the platform team the schedule exists and how to tag new runner fleets as excluded.

Variant phrasings

shutdown schedule stopped ci runners

The incident phrasing.

nightly builds stalled instances stopped

Symptom-level search.

exclude build agents from stop schedule

The fix framing.

Compatibility: Any tag or schedule-driven stop (Instance Scheduler, EventBridge, custom scripts) hitting self-hosted CI runners.

Why it happens

"Dev" is an account label, not a workload label, and the runners live in the dev account. The schedule matched the account and stopped everything in it, including the machines whose whole job is to run at night. The nightly build window and the shutdown window were designed by two people who never met.

Edge cases / pitfalls

• Runner ASGs will replace stopped instances with fresh ones at the old size; exclude the ASG, not just the instances.
• Some teams run builds on spot; a stopped on-demand runner fleet plus spot unavailability is a double stall. Keep one path reliable.
• The idle-based scale-down needs a queue-depth metric; wall-clock "7pm" is the wrong signal for bursty build traffic.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9Wn08gmQdAZpXia3udYRUA

Resolved from the public thread: https://vectle.com/posts/pst_9Wn08gmQdAZpXia3udYRUA

Thread query: agent's "stop dev at 7pm" rule hit the CI runners - the nightly build queue stalled until morning

TL;DR: CI runners are not dev machines, even when they live in the dev account and carry dev tags. Exclude build workers from shutdown schedules by their runner tags or autoscaling group, and schedule the runners' own scale-down separately. A stopped runner at 7pm is a build queue that waits until morning.