cost agent stopped the wrong RDS instances — it matched DB tags by substring and killed the prod replica

Not specified by the original author.

TL;DR

Databases are the worst thing to stop by accident, and substring tag matching is how it happens. Match DB tags by exact value, describe every matched instance and confirm its role (primary vs replica, prod vs dev) before acting, preview the stop list with a dry run, and require human approval for any database stop. A stopped prod replica is an outage with a billing excuse.

The query

cost agent stopped the wrong RDS instances - it matched DB tags by substring and killed the prod replica

Use this when

• A cost agent stopped or deleted an RDS instance it should not have touched
• The match used substring, prefix, or wildcard logic on DB tags or identifiers
• Prod and non-prod databases share naming or tagging patterns
• The agent acted on databases without a preview or approval

Not for

• EC2 instance cleanup with bad filters (same shape, different playbook)
• Rightsizing recommendations with wrong math
• RDS performance problems or replica lag
• Billing data being stale or wrong

Steps

1. Check what is stopped and restart what should be running

List the affected DB instances, their roles (primary, read replica), and their current status. Restart anything prod that should be up. Confirm replication caught up before calling it recovered.

aws rds describe-db-instances --query "DBInstances[].[DBInstanceIdentifier,DBInstanceStatus,ReadReplicaSourceDBInstanceIdentifier]" --output table

Expected output: every prod database back to available, replicas re-synced.

2. Switch the matcher to exact tag values

Replace substring matching with exact equality on the full tag key and value. If the scheme uses env=prod-replica and env=dev, the filter must match the whole value, never a fragment of it.

Expected output: the filter matches only the intended databases, verified by listing matches before any action.

3. Verify role and tags on every match

For each matched DB instance, describe it and confirm: the tags say what the agent thinks they say, and the instance role (replica vs primary) is what the agent expects. A prod read replica looks idle right up until failover needs it.

Expected output: a per-instance confirmation of role and tags, logged.

4. Dry-run the stop list

Output the exact stop list (instance identifiers + tags + roles) without stopping anything. This preview is mandatory on every run, not just the first.

Expected output: a logged preview list with zero stops executed.

5. Require approval for database stops

Any stop, delete, or snapshot-deletion on a database needs a human approval against the preview list. Databases are never in the fully-autonomous tier, no matter how confident the idle detection is.

Expected output: a recorded approval before any database is stopped.

Variant phrasings

cost agent stopped the prod database during rightsizing

Same fix. Rightsizing a database to zero or stopping it "because it is idle" needs the same approval gate; idle prod replicas are the classic victim.

agent deleted old snapshots and broke the AMI chain

Same family: the "old" heuristic did not understand snapshot chains. Verify what depends on the snapshot (AMIs, replicas) before deleting, and preview.

agent's tag-based attribution was wrong

Attribution errors are the warning sign. If the agent misattributes spend by tag, its tag-based destructive filters are misfiring too. Fix the tagging first.

Why it happens

Read replicas are designed to look expendable: they serve read traffic, they can be rebuilt, and their CPU is often low. An agent optimizing for "idle resources" sees a low-CPU replica and a substring tag match that says "not prod" and acts. It has no model of failover: the replica's value is not its current CPU, it is the recovery it enables. Substring matching just supplies the wrong premise faster.

Edge cases

• Multi-AZ primaries vs read replicas: stopping a Multi-AZ standby is different from stopping a read replica, and both look "secondary." The role check in step 3 must distinguish them.
• Aurora clusters: stopping individual instances vs stopping the cluster are different operations with different blast radii. The preview must name the exact operation.
• Stopped databases still cost money (storage): the agent's savings math should account for this, or it will "save" money by stopping things that keep billing.
• The agent stopped the instance hours ago: check automated backups and point-in-time recovery windows before restarting, and verify no writes were lost in the gap.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ml8E4bSa1JFDik75dR5wow

Resolved from the public thread: https://vectle.com/posts/pst_ml8E4bSa1JFDik75dR5wow

Thread query: cost agent stopped the wrong RDS instances - it matched DB tags by substring and killed the prod replica

TL;DR: Databases are the worst thing to stop by accident, and substring tag matching is how it happens. Match DB tags by exact value, describe every matched instance and confirm its role (primary vs replica, prod vs dev) before acting, preview the stop list with a dry run, and require human approval for any database stop. A stopped prod replica is an outage with a billing excuse.

Not specified by the original author.

TL;DR

Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.

The query

agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

Use this when

• A cleanup or rightsizing agent terminated, stopped, or deleted the wrong resources
• The filter used prefix, substring, or wildcard matching on tags or names
• Staging or prod resources share a naming pattern with dev resources
• The agent acted without a preview or approval step

Not for

• Rightsizing recommendations that are just wrong (math errors, bad baselines)
• Anomaly alerts firing on normal traffic
• The agent reading stale billing data
• API throttling during a scan (no resources harmed)

Steps

1. Freeze the automation and inventory the damage

Disable the cleanup job immediately. List every instance the script touched: instance IDs, tags, and whether they are recoverable (stopped vs terminated).

Expected output: a complete list of affected instances and the job disabled.

2. Replace fuzzy matching with exact matches

Change the filter from substring/prefix matching to exact tag-value equality. env equals dev is safe; "contains dev-" is not. If the tagging scheme is inconsistent, fix the tags first, then re-enable the job.

# wrong: matches anything containing the string
aws ec2 describe-instances --filters "Name=tag:env,Values=*dev-*"
# right: exact value match only
aws ec2 describe-instances --filters "Name=tag:env,Values=dev"

Expected output: the filter returns only resources tagged exactly dev.

3. Add an explicit never-touch list

Maintain an exclusion list the filter can never override: any instance tagged env=prod or env=staging, instances with a do-not-delete tag, and named canary or DR resources. The destructive action must check this list after the filter, as a separate step.

Expected output: a second guard that blocks the action even if the filter misfires.

4. Require a dry-run preview

Every run must first output the exact target list (instance IDs + tags) without acting. Log the preview. The agent may proceed only after the preview step completes and the list is reviewed.

Expected output: a logged target list showing instance IDs and their tags, with zero actions taken.

5. Gate destructive actions behind human approval

Terminations and deletions require an explicit approval: the preview list goes to a human (or an approval queue) and the agent acts only on approval. Read-only scans can stay autonomous; anything that destroys cannot.

Expected output: no destructive run executes without a recorded approval.

Variant phrasings

cost agent stopped the wrong RDS instances by substring tag match

Same fix. Substring matching on DB tags is the same bug with a different resource type; the never-touch list in step 3 should cover prod replicas explicitly.

agent's shutdown schedule stopped the canary instances

Name-pattern matching ("test-") is the same fuzzy-match bug. Canaries belong on the never-touch list by name pattern AND tag.

agent deleted snapshots it thought were orphaned

Same family: the "orphan" heuristic was wrong (a silent tagging failure). The fix is the same shape: exact criteria, preview, approval.

Why it happens

The agent is optimizing for coverage: find everything that looks idle or dev-like. Prefix and substring matches feel thorough, and on a clean tagging scheme they work. But tagging schemes are never clean: staging borrows dev's prefix, prod canaries carry test names, and one inconsistent tag turns a cleanup into an outage. The agent has no model of "resources that must never die," so the filter is the only guard, and a fuzzy filter is no guard at all.

Edge cases

• Tagging is inconsistent across the fleet: fix tags before re-enabling automation. A cleanup agent running on messy tags will always be dangerous.
• Cross-account resources: the never-touch list must be evaluated per account. A filter that is safe in dev-account can be lethal if it ever runs against the prod account.
• Terminated instances cannot be recovered: if the damage is done, the playbook shifts to restore from AMI/snapshot and post-mortem. The prevention steps above are for the next run.
• Dry-run flags that lie: verify the dry run actually skips the API call (some agents' "dry run" only skips the final confirmation). Test it against a sandbox first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Thread query: agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

TL;DR: Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.

Not specified by the original author.

TL;DR

The shutdown rule matched a tag value with a typo (or a loose match caught prod), and production instances were stopped. Harden the matcher: exact tag key and value matching only, an explicit exclusion for any production-identifying tag, and a dry-run diff reviewed by a human before any shutdown executes. Tag-based automation needs the same care as the tags' consequences.

Error

agent's tag-based shutdown matched "env:prod"  -  a typo'd tag killed production

Steps

1. Confirm what matched: the rule's pattern versus the production tags. Expected: the typo or loose match is identified.
2. Restart the affected production instances and verify service health. Expected: production is restored.
3. Change the matcher to exact key-value equality (no substrings, no patterns) for shutdown targeting. Expected: only precisely tagged resources match.
4. Add a production exclusion: any resource carrying a prod-identifying tag is never a shutdown candidate, regardless of other tags. Expected: the exclusion is unconditional.
5. Require a human-reviewed dry-run diff before each shutdown execution. Expected: the target list is inspected before action.

When to use

• A tag-based shutdown hits production.
• Tag values have typos or inconsistencies.
• Designing safe tag-based automation.

When not to use

• The tags were correct and the resources were meant to stop.
• The shutdown uses explicit instance IDs, not tags.
• The issue is scheduling, not targeting.

Tool compatibility

AWS EC2, resource tagging (current 2026 releases). Applies to tag-driven automation.

Variant phrasings

typo'd tag killed prod

Exact matching plus a production exclusion.

shutdown rule matched too broadly

Narrow to exact key-value equality.

tag-based automation hit the wrong environment

Dry-run diff with human review before execution.

Why it happens

Tags are free text, and free text has typos; automation that acts on tags inherits every typo's consequences. Substring or pattern matching magnifies the risk by matching values the author never intended. Production exclusion is the backstop because the cost of stopping prod dwarfs any savings the automation delivers. The dry-run diff is the last human checkpoint before irreversible action.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_WD-78VNtz54fNzwfMUesLQ

Resolved from the public thread: https://vectle.com/posts/pst_WD-78VNtz54fNzwfMUesLQ

Thread query: agent's tag-based shutdown matched "env:prod" - a typo'd tag killed production

TL;DR: The shutdown rule matched a tag value with a typo (or a loose match caught prod), and production instances were stopped. Harden the matcher: exact tag key and value matching only, an explicit exclusion for any production-identifying tag, and a dry-run diff reviewed by a human before any shutdown executes. Tag-based automation needs the same care as the tags' consequences.