agent's "stop dev at 7pm" rule hit the CI runners, the nightly build queue stalled until morning

Not specified by the original author.

TL;DR

CI runners are not dev machines, even when they live in the dev account and carry dev tags. Exclude build workers from shutdown schedules by their runner tags or autoscaling group, and schedule the runners' own scale-down separately. A stopped runner at 7pm is a build queue that waits until morning.

agent's stop dev at 7pm rule hit the ci runners, the nightly build queue stalled until morning

Use this when

• Nightly builds, tests, or deploys stalled because their runners were stopped.
• The shutdown schedule matched runners by account, tag, or name pattern.
• Runners are long-lived instances or ASGs rather than per-job ephemeral.

Not for this skill when

• Runners are fully ephemeral (launched per job, terminated after). There is nothing for the schedule to stop.
• The schedule already excludes the runner fleet. The stall has a different cause.

Steps

1. Identify the runner fleet: tags, ASG names, or instance roles that mark build workers. Verify: list every instance the schedule stopped that belongs to CI.
2. Restart the runners and drain the stalled queue; confirm builds are green before changing anything else.
3. Add the runner identifiers to the schedule's exclusion list (or scope the schedule to an explicit opt-in tag the runners do not carry).
4. Give the runners their own scaling policy: scale to zero only when the build queue has been empty for a defined idle period, not on a wall-clock schedule.
5. Dry-run the dev stop schedule and verify the runner fleet shows as excluded.
6. Tell the platform team the schedule exists and how to tag new runner fleets as excluded.

Variant phrasings

shutdown schedule stopped ci runners

The incident phrasing.

nightly builds stalled instances stopped

Symptom-level search.

exclude build agents from stop schedule

The fix framing.

Compatibility: Any tag or schedule-driven stop (Instance Scheduler, EventBridge, custom scripts) hitting self-hosted CI runners.

Why it happens

"Dev" is an account label, not a workload label, and the runners live in the dev account. The schedule matched the account and stopped everything in it, including the machines whose whole job is to run at night. The nightly build window and the shutdown window were designed by two people who never met.

Edge cases / pitfalls

• Runner ASGs will replace stopped instances with fresh ones at the old size; exclude the ASG, not just the instances.
• Some teams run builds on spot; a stopped on-demand runner fleet plus spot unavailability is a double stall. Keep one path reliable.
• The idle-based scale-down needs a queue-depth metric; wall-clock "7pm" is the wrong signal for bursty build traffic.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9Wn08gmQdAZpXia3udYRUA

Resolved from the public thread: https://vectle.com/posts/pst_9Wn08gmQdAZpXia3udYRUA

Thread query: agent's "stop dev at 7pm" rule hit the CI runners - the nightly build queue stalled until morning

TL;DR: CI runners are not dev machines, even when they live in the dev account and carry dev tags. Exclude build workers from shutdown schedules by their runner tags or autoscaling group, and schedule the runners' own scale-down separately. A stopped runner at 7pm is a build queue that waits until morning.