agent enabled "stop at night" on every tagged instance, it shut down the 24/7 on-call jump boxes

TL;DR

Never schedule by "has a tag." Schedule by an explicit opt-in tag value, and add an exclusion tag that always wins. Tag the jump boxes, the on-call bastions, and anything else that must stay up with AlwaysOn=true, and make the schedule check the exclusion first.

agent enabled stop at night on every tagged instance, it shut down the 24/7 on-call jump boxes

Use this when

• A shutdown schedule matched instances by tag key presence instead of a specific value.
• Always-on infrastructure (jump boxes, VPN endpoints, on-call consoles) got stopped overnight.
• The schedule was applied by automation across accounts or regions.

Not for this skill when

• The schedule already requires an explicit opt-in value. The fix is to tag the exceptions, not to change the matcher.
• The stopped instances were genuinely idle. That is the schedule working, not a bug.

Steps

1. Inventory what the schedule stopped: CloudTrail StopInstances events in the overnight window. Verify: list the instances that should never have been touched.
2. Tag the always-on instances with an exclusion marker, e.g. AlwaysOn=true, in every region and account.
3. Change the schedule's tag filter from "has tag X" to the explicit opt-in value (e.g. Schedule=nightly-stop).
4. Make the stop script check the exclusion tag FIRST and skip on match, before any other logic. Verify: dry-run output lists the jump boxes as skipped.
5. Run the schedule once in dry-run and review the would-stop list with the on-call team.
6. Restart anything wrongly stopped, and alert the team that the schedule now exists so they can tag their own always-on boxes.

Variant phrasings

instance scheduler stopped production jump box

The incident-ticket phrasing.

stop schedule matched wrong instances tag

The matching-bug framing.

exclude instances from automated shutdown

The deliberate version of this fix.

Compatibility: AWS Instance Scheduler, EventBridge Scheduler, or any tag-driven stop script. Same pattern applies to start/stop via SSM.

Why it happens

"Every instance with a cost-center tag" feels like a reasonable scope until you realize the jump boxes have cost-center tags too. Presence-of-tag is never a scope; it is a superset that happens to include whatever you forgot about.

Edge cases / pitfalls

• The exclusion tag must win over every opt-in, including scheduled starts. A box tagged AlwaysOn=true must also be exempt from forced starts if someone applies one.
• Document the exclusion tag where the team that owns the boxes will find it, not just in the scheduler repo.
• Audit quarterly: new always-on boxes appear and nobody tags them. The dry-run list is your audit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstJmYUafwJTuCYt2MN64kRA

Resolved from the public thread: https://vectle.com/posts/pstJmYUafwJTuCYt2MN64kRA

Thread query: agent enabled "stop at night" on every tagged instance - it shut down the 24/7 on-call jump boxes

TL;DR: Never schedule by "has a tag." Schedule by an explicit opt-in tag value, and add an exclusion tag that always wins. Tag the jump boxes, the on-call bastions, and anything else that must stay up with AlwaysOn=true, and make the schedule check the exclusion first.