agent's tag-based shutdown matched "env:prod" - a typo'd tag killed production
Fixes a tag-based shutdown that kills production on a tag typo by requiring exact-match validation and a production exclusion. Use when a tag typo causes a shutdown agent to target production. Not for correctly tagged resources.
TL;DR
The shutdown rule matched a tag value with a typo (or a loose match caught prod), and production instances were stopped. Harden the matcher: exact tag key and value matching only, an explicit exclusion for any production-identifying tag, and a dry-run diff reviewed by a human before any shutdown executes. Tag-based automation needs the same care as the tags' consequences.
Error
agent's tag-based shutdown matched "env:prod" - a typo'd tag killed productionSteps
- Confirm what matched: the rule's pattern versus the production tags. Expected: the typo or loose match is identified.
- Restart the affected production instances and verify service health. Expected: production is restored.
- Change the matcher to exact key-value equality (no substrings, no patterns) for shutdown targeting. Expected: only precisely tagged resources match.
- Add a production exclusion: any resource carrying a prod-identifying tag is never a shutdown candidate, regardless of other tags. Expected: the exclusion is unconditional.
- Require a human-reviewed dry-run diff before each shutdown execution. Expected: the target list is inspected before action.
When to use
- A tag-based shutdown hits production.
- Tag values have typos or inconsistencies.
- Designing safe tag-based automation.
When not to use
- The tags were correct and the resources were meant to stop.
- The shutdown uses explicit instance IDs, not tags.
- The issue is scheduling, not targeting.
Tool compatibility
AWS EC2, resource tagging (current 2026 releases). Applies to tag-driven automation.
Variant phrasings
typo'd tag killed prod
Exact matching plus a production exclusion.
shutdown rule matched too broadly
Narrow to exact key-value equality.
tag-based automation hit the wrong environment
Dry-run diff with human review before execution.
Why it happens
Tags are free text, and free text has typos; automation that acts on tags inherits every typo's consequences. Substring or pattern matching magnifies the risk by matching values the author never intended. Production exclusion is the backstop because the cost of stopping prod dwarfs any savings the automation delivers. The dry-run diff is the last human checkpoint before irreversible action.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_WD-78VNtz54fNzwfMUesLQ