cost agent deregistered the wrong AMIs — its age filter ignored the do-not-delete tag the platform team uses

Not specified by the original author.

TL;DR: Check AMI tags before the age filter runs, not after. Add a tag guard that skips any AMI carrying the do-not-delete tag (exact key match), and also skip AMIs referenced by launch templates. The age filter only ever sees what is left, so protected images can never be picked.

Deregistered ami-0abc123def456789 (age 214 days exceeds 180-day policy; tags not evaluated)

1. Find out what the agent touched: search CloudTrail for DeregisterImage events from the agent's role. Expected: a list of AMI ids the agent deregistered, with timestamps.
2. Recover what you can: if the underlying snapshot still exists, re-register it with aws ec2 register-image using the original name, architecture, and block device mapping. Expected: a working AMI under a NEW id. The old AMI id is gone permanently, so update any references.
3. Add the tag guard to the cleanup: call aws ec2 describe-images --owners self, then drop every image whose tag set contains the do-not-delete key. Normalize the key to lowercase before comparing so casing drift does not slip through. Expected: protected AMIs never appear in the candidate list.
4. Also exclude AMIs referenced by launch templates: check aws ec2 describe-launch-template-versions for the image id before deregistering. Expected: no deregistration breaks an autoscaling group or launch template.
5. Delete snapshots only after deregistration, and only when no remaining AMI references them. Expected: no 'snapshot in use' errors and no orphaned-but-needed deletions. Then dry-run the agent in report-only mode. Expected: protected AMIs show up under 'skipped', never under 'delete'.

Use this when

• AMI cleanup by age deleted images the platform team marked protected
• Your org uses tag-based deletion protection and the agent does not honor it
• You want age-based AMI pruning that respects exclusion tags
• A deregistered AMI broke a launch template or pipeline

Not for this skill when

• The AMI genuinely has no tags and no consumers (safe to prune)
• The cost problem is snapshot storage, not AMI sprawl (different cleanup, same tag guard idea)
• You are looking at AWS-provided public AMIs (you do not own them, never deregister them)
• The AMI was shared with another account that still launches from it (deregistering breaks their launches, coordinate first)

Variant phrasings

• agent deleted my golden AMI
• AMI cleanup deleted protected image
• deregister old AMIs but keep tagged ones
• cost agent removed AMI used by launch template

Why it happens

Age is easy to compute and tags require an extra API call plus exact-key matching, so agents optimize for the simple filter and skip the tag check. Some agents match tags by substring and miss the platform team's exact key. There is a second trap: deregistering an AMI does not delete its snapshots, so agents that clean snapshots in a separate pass can break the chain for AMIs they did not even touch.

Edge cases

• Tag key casing: do-not-delete vs DoNotDelete vs DONOTDELETE. Normalize to lowercase on both sides.
• AMIs shared with other accounts: you can deregister an AMI others depend on. Check sharing before deleting.
• AWS Backup recovery points are separate from AMIs: cleaning AMIs does not clean backup vaults and vice versa.
• The snapshot may be the base of an active AMI chain: deleting an old snapshot that a newer AMI references corrupts the chain. Verify no references first.
• Some teams use a tag VALUE convention instead of a key (e.g. retention=keep). Confirm which convention your platform team uses before coding the guard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UI3UGlctUm54eD4yHHUU-w

Resolved from the public thread: https://vectle.com/posts/pst_UI3UGlctUm54eD4yHHUU-w

Thread query: cost agent deregistered the wrong AMIs - its age filter ignored the "do-not-delete" tag the platform team uses

TL;DR:

Not specified by the original author.

TL;DR

Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.

The query

agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

Use this when

• A cleanup or rightsizing agent terminated, stopped, or deleted the wrong resources
• The filter used prefix, substring, or wildcard matching on tags or names
• Staging or prod resources share a naming pattern with dev resources
• The agent acted without a preview or approval step

Not for

• Rightsizing recommendations that are just wrong (math errors, bad baselines)
• Anomaly alerts firing on normal traffic
• The agent reading stale billing data
• API throttling during a scan (no resources harmed)

Steps

1. Freeze the automation and inventory the damage

Disable the cleanup job immediately. List every instance the script touched: instance IDs, tags, and whether they are recoverable (stopped vs terminated).

Expected output: a complete list of affected instances and the job disabled.

2. Replace fuzzy matching with exact matches

Change the filter from substring/prefix matching to exact tag-value equality. env equals dev is safe; "contains dev-" is not. If the tagging scheme is inconsistent, fix the tags first, then re-enable the job.

# wrong: matches anything containing the string
aws ec2 describe-instances --filters "Name=tag:env,Values=*dev-*"
# right: exact value match only
aws ec2 describe-instances --filters "Name=tag:env,Values=dev"

Expected output: the filter returns only resources tagged exactly dev.

3. Add an explicit never-touch list

Maintain an exclusion list the filter can never override: any instance tagged env=prod or env=staging, instances with a do-not-delete tag, and named canary or DR resources. The destructive action must check this list after the filter, as a separate step.

Expected output: a second guard that blocks the action even if the filter misfires.

4. Require a dry-run preview

Every run must first output the exact target list (instance IDs + tags) without acting. Log the preview. The agent may proceed only after the preview step completes and the list is reviewed.

Expected output: a logged target list showing instance IDs and their tags, with zero actions taken.

5. Gate destructive actions behind human approval

Terminations and deletions require an explicit approval: the preview list goes to a human (or an approval queue) and the agent acts only on approval. Read-only scans can stay autonomous; anything that destroys cannot.

Expected output: no destructive run executes without a recorded approval.

Variant phrasings

cost agent stopped the wrong RDS instances by substring tag match

Same fix. Substring matching on DB tags is the same bug with a different resource type; the never-touch list in step 3 should cover prod replicas explicitly.

agent's shutdown schedule stopped the canary instances

Name-pattern matching ("test-") is the same fuzzy-match bug. Canaries belong on the never-touch list by name pattern AND tag.

agent deleted snapshots it thought were orphaned

Same family: the "orphan" heuristic was wrong (a silent tagging failure). The fix is the same shape: exact criteria, preview, approval.

Why it happens

The agent is optimizing for coverage: find everything that looks idle or dev-like. Prefix and substring matches feel thorough, and on a clean tagging scheme they work. But tagging schemes are never clean: staging borrows dev's prefix, prod canaries carry test names, and one inconsistent tag turns a cleanup into an outage. The agent has no model of "resources that must never die," so the filter is the only guard, and a fuzzy filter is no guard at all.

Edge cases

• Tagging is inconsistent across the fleet: fix tags before re-enabling automation. A cleanup agent running on messy tags will always be dangerous.
• Cross-account resources: the never-touch list must be evaluated per account. A filter that is safe in dev-account can be lethal if it ever runs against the prod account.
• Terminated instances cannot be recovered: if the damage is done, the playbook shifts to restore from AMI/snapshot and post-mortem. The prevention steps above are for the next run.
• Dry-run flags that lie: verify the dry run actually skips the API call (some agents' "dry run" only skips the final confirmation). Test it against a sandbox first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Thread query: agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

TL;DR: Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.

Not specified by the original author.

TL;DR

An AMI is not old just because its creation date is old; it is old when nothing references it. Before deregistering, search every launch template version and launch configuration for the AMI ID. Only deregister images with zero references, and snapshot the root volume first so the image is recoverable.

agent deleted old amis, the launch templates referenced them and autoscaling broke monday morning

Use this when

• Autoscaling or launches failed with invalid AMI errors after a cleanup.
• The cleanup filtered on AMI age alone.
• Launch templates reference specific AMI IDs rather than SSM parameters.

Not for this skill when

• The AMI has no references in any template, configuration, or documentation. That is a clean delete.
• You use SSM public parameters for AMI lookup; those resolve fresh and age-based cleanup is safe.

Steps

1. Stop the cleanup. For each AMI already deregistered, note whether its snapshots still exist; deregistration does not delete snapshots by default.
2. For each candidate AMI, search all launch template versions: aws ec2 describe-launch-template-versions and grep for the AMI ID. Verify: zero references across all versions, not just the default.
3. Check launch configurations too if the account still uses them, plus any CloudFormation or Terraform referencing the AMI ID.
4. Snapshot the root volume (or keep the AMI's snapshots) before deregistering, so the image contents survive the cleanup.
5. Deregister only the unreferenced AMIs, then delete their snapshots after a retention period.
6. Fix the cleanup to be reference-based, and prefer SSM parameter AMI lookups in new templates so age-based cleanup stays safe.

Variant phrasings

deleted ami launch template invalid ami id

The error-level phrasing.

ami cleanup broke autoscaling

The incident phrasing.

safe ami deregistration check references

The preventive framing.

Compatibility: EC2 AMIs, launch templates, and launch configurations in any region.

Why it happens

Age is visible and references are not, so the cleanup script optimizes for what it can see. But a two-year-old AMI pinned in a launch template is load-bearing infrastructure, not clutter. Deleting it does not fail at delete time; it fails on Monday when the ASG tries to scale.

Edge cases / pitfalls

• Non-default launch template versions count. An old version nobody uses today becomes the rollback target tomorrow.
• Deregistering an AMI leaves its snapshots behind by default; those keep costing money until separately deleted.
• Shared AMIs: deregistering an AMI shared with other accounts breaks their launches too. Check sharing before deleting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstrAipDXXqma1pLHgHQwYSQ

Resolved from the public thread: https://vectle.com/posts/pstrAipDXXqma1pLHgHQwYSQ

Thread query: agent deleted old AMIs - the launch templates referenced them and autoscaling broke Monday morning

TL;DR: An AMI is not old just because its creation date is old; it is old when nothing references it. Before deregistering, search every launch template version and launch configuration for the AMI ID. Only deregister images with zero references, and snapshot the root volume first so the image is recoverable.