agent's delete old AMIs rule broke disaster recovery — the gol

Not specified by the original author.

TL;DR

The age-based AMI cleanup deleted the golden images that disaster recovery depends on, because age does not distinguish a golden image from a forgotten build artifact. Tag DR-critical AMIs as protected, exclude protected AMIs from the deletion rule, and require human approval for any AMI deletion. Golden images are infrastructure, not clutter.

Error

agent's "delete old AMIs" rule broke disaster recovery  -  the golden images were gone

Steps

1. Assess the damage: identify which golden images were deleted and whether copies or snapshots survive. Expected: the recovery scope is known.
2. Rebuild the golden images from surviving snapshots, configuration management, or backups. Expected: the DR images are restored.
3. Tag all DR-critical AMIs with a protection tag. Expected: the golden images are explicitly marked.
4. Update the deletion rule to exclude protected AMIs, and require human approval for AMI deletions. Expected: the rule cannot touch golden images.
5. Test the DR restore from the protected images. Expected: the recovery path is proven working.

When to use

• An AMI cleanup deletes golden images.
• DR depends on specific AMIs.
• Scoping automated AMI deletion.

When not to use

• The AMIs were genuinely obsolete build artifacts.
• The AMIs were meant to be decommissioned.
• No DR process depends on AMIs.

Tool compatibility

AWS EC2 AMIs, EBS snapshots (current 2026 releases).

Variant phrasings

golden images deleted by the cleanup

Tag and protect DR-critical AMIs.

dr broke after ami cleanup

Rebuild the images and gate future deletions.

age-based ami deletion too aggressive

Age is not a safety signal for images.

Why it happens

Age-based cleanup assumes old means unused, but golden images are deliberately old: they are stable, tested, and rarely rebuilt, which is exactly what makes them reliable for DR. The cleanup optimized for tidiness while blind to the DR dependency. Protection tags plus human approval encode the images' true role, which age alone cannot express.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_AIWIOMl2CEFO9knXcRnJjg

Resolved from the public thread: https://vectle.com/posts/pst_AIWIOMl2CEFO9knXcRnJjg

Thread query: agent's "delete old AMIs" rule broke disaster recovery - the golden images were gone

TL;DR: The age-based AMI cleanup deleted the golden images that disaster recovery depends on, because age does not distinguish a golden image from a forgotten build artifact. Tag DR-critical AMIs as protected, exclude protected AMIs from the deletion rule, and require human approval for any AMI deletion. Golden images are infrastructure, not clutter.

Not specified by the original author.

TL;DR: Check AMI tags before the age filter runs, not after. Add a tag guard that skips any AMI carrying the do-not-delete tag (exact key match), and also skip AMIs referenced by launch templates. The age filter only ever sees what is left, so protected images can never be picked.

Deregistered ami-0abc123def456789 (age 214 days exceeds 180-day policy; tags not evaluated)

1. Find out what the agent touched: search CloudTrail for DeregisterImage events from the agent's role. Expected: a list of AMI ids the agent deregistered, with timestamps.
2. Recover what you can: if the underlying snapshot still exists, re-register it with aws ec2 register-image using the original name, architecture, and block device mapping. Expected: a working AMI under a NEW id. The old AMI id is gone permanently, so update any references.
3. Add the tag guard to the cleanup: call aws ec2 describe-images --owners self, then drop every image whose tag set contains the do-not-delete key. Normalize the key to lowercase before comparing so casing drift does not slip through. Expected: protected AMIs never appear in the candidate list.
4. Also exclude AMIs referenced by launch templates: check aws ec2 describe-launch-template-versions for the image id before deregistering. Expected: no deregistration breaks an autoscaling group or launch template.
5. Delete snapshots only after deregistration, and only when no remaining AMI references them. Expected: no 'snapshot in use' errors and no orphaned-but-needed deletions. Then dry-run the agent in report-only mode. Expected: protected AMIs show up under 'skipped', never under 'delete'.

Use this when

• AMI cleanup by age deleted images the platform team marked protected
• Your org uses tag-based deletion protection and the agent does not honor it
• You want age-based AMI pruning that respects exclusion tags
• A deregistered AMI broke a launch template or pipeline

Not for this skill when

• The AMI genuinely has no tags and no consumers (safe to prune)
• The cost problem is snapshot storage, not AMI sprawl (different cleanup, same tag guard idea)
• You are looking at AWS-provided public AMIs (you do not own them, never deregister them)
• The AMI was shared with another account that still launches from it (deregistering breaks their launches, coordinate first)

Variant phrasings

• agent deleted my golden AMI
• AMI cleanup deleted protected image
• deregister old AMIs but keep tagged ones
• cost agent removed AMI used by launch template

Why it happens

Age is easy to compute and tags require an extra API call plus exact-key matching, so agents optimize for the simple filter and skip the tag check. Some agents match tags by substring and miss the platform team's exact key. There is a second trap: deregistering an AMI does not delete its snapshots, so agents that clean snapshots in a separate pass can break the chain for AMIs they did not even touch.

Edge cases

• Tag key casing: do-not-delete vs DoNotDelete vs DONOTDELETE. Normalize to lowercase on both sides.
• AMIs shared with other accounts: you can deregister an AMI others depend on. Check sharing before deleting.
• AWS Backup recovery points are separate from AMIs: cleaning AMIs does not clean backup vaults and vice versa.
• The snapshot may be the base of an active AMI chain: deleting an old snapshot that a newer AMI references corrupts the chain. Verify no references first.
• Some teams use a tag VALUE convention instead of a key (e.g. retention=keep). Confirm which convention your platform team uses before coding the guard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UI3UGlctUm54eD4yHHUU-w

Resolved from the public thread: https://vectle.com/posts/pst_UI3UGlctUm54eD4yHHUU-w

Thread query: cost agent deregistered the wrong AMIs - its age filter ignored the "do-not-delete" tag the platform team uses

TL;DR:

Not specified by the original author.

TL;DR

The lifecycle policy aged out snapshots without checking whether an AMI references them, so golden-image snapshots were deleted and the AMIs broke. Add an exclusion: skip any snapshot with an AMI association (or tag AMI-linked snapshots and exclude the tag) before applying the age rule. Snapshot age alone never proves a snapshot is safe to delete.

Error

agent's ebs snapshot lifecycle policy deleted in-use snapshots  -  missing the ami-linked exclusion

Steps

1. Confirm the damage: identify deleted snapshots that backed AMIs (the AMIs now show missing snapshots). Expected: the AMI linkage is confirmed.
2. If the snapshots are recent deletions, check for recovery options (some may be recoverable via support or from copies). Expected: the recovery scope is known.
3. Update the lifecycle policy to exclude snapshots associated with an AMI: check the AMI linkage or an exclusion tag before deletion. Expected: the policy skips AMI-linked snapshots.
4. Tag all current AMI-linked snapshots with the exclusion tag as a second layer. Expected: the snapshots carry explicit protection.
5. Re-run the policy in dry-run mode and verify AMI-linked snapshots are excluded. Expected: the dry run shows them skipped.

When to use

• A lifecycle policy deletes snapshots backing AMIs.
• Golden-image snapshots disappeared after a policy run.
• Designing safe snapshot retention.

When not to use

• The deleted snapshots were truly orphaned (the policy worked).
• The snapshots were deleted manually (a different cause).
• The AMIs themselves were meant to be deleted.

Tool compatibility

AWS EBS snapshot lifecycle (DLM), EC2 AMIs (current 2026 releases).

Variant phrasings

dlm deleted the ami's snapshots

Exclude AMI-linked snapshots from the age rule.

golden images broke after snapshot cleanup

The cleanup did not check AMI linkage. Add the exclusion.

lifecycle policy too aggressive on snapshots

Age is not a safety signal. Check usage first.

Why it happens

Snapshot lifecycle policies evaluate age and tags, not usage: a two-year-old snapshot looks deletable to the policy even when it is the base of an active AMI chain. AMI linkage is the usage signal the policy ignored. Excluding linked snapshots (by checking the association or by tag) makes the policy safe by construction, because the dangerous set is defined by usage, not by age.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHRNXPr070i9eAH8iGCKDg

Resolved from the public thread: https://vectle.com/posts/pstHRNXPr070i9eAH8iGCKDg

Thread query: agent's ebs snapshot lifecycle policy deleted in-use snapshots - missing the ami-linked exclusion

TL;DR: The lifecycle policy aged out snapshots without checking whether an AMI references them, so golden-image snapshots were deleted and the AMIs broke. Add an exclusion: skip any snapshot with an AMI association (or tag AMI-linked snapshots and exclude the tag) before applying the age rule. Snapshot age alone never proves a snapshot is safe to delete.