agent's ebs snapshot lifecycle policy deleted in-use snapshots — m

Not specified by the original author.

TL;DR

The lifecycle policy aged out snapshots without checking whether an AMI references them, so golden-image snapshots were deleted and the AMIs broke. Add an exclusion: skip any snapshot with an AMI association (or tag AMI-linked snapshots and exclude the tag) before applying the age rule. Snapshot age alone never proves a snapshot is safe to delete.

Error

agent's ebs snapshot lifecycle policy deleted in-use snapshots  -  missing the ami-linked exclusion

Steps

1. Confirm the damage: identify deleted snapshots that backed AMIs (the AMIs now show missing snapshots). Expected: the AMI linkage is confirmed.
2. If the snapshots are recent deletions, check for recovery options (some may be recoverable via support or from copies). Expected: the recovery scope is known.
3. Update the lifecycle policy to exclude snapshots associated with an AMI: check the AMI linkage or an exclusion tag before deletion. Expected: the policy skips AMI-linked snapshots.
4. Tag all current AMI-linked snapshots with the exclusion tag as a second layer. Expected: the snapshots carry explicit protection.
5. Re-run the policy in dry-run mode and verify AMI-linked snapshots are excluded. Expected: the dry run shows them skipped.

When to use

• A lifecycle policy deletes snapshots backing AMIs.
• Golden-image snapshots disappeared after a policy run.
• Designing safe snapshot retention.

When not to use

• The deleted snapshots were truly orphaned (the policy worked).
• The snapshots were deleted manually (a different cause).
• The AMIs themselves were meant to be deleted.

Tool compatibility

AWS EBS snapshot lifecycle (DLM), EC2 AMIs (current 2026 releases).

Variant phrasings

dlm deleted the ami's snapshots

Exclude AMI-linked snapshots from the age rule.

golden images broke after snapshot cleanup

The cleanup did not check AMI linkage. Add the exclusion.

lifecycle policy too aggressive on snapshots

Age is not a safety signal. Check usage first.

Why it happens

Snapshot lifecycle policies evaluate age and tags, not usage: a two-year-old snapshot looks deletable to the policy even when it is the base of an active AMI chain. AMI linkage is the usage signal the policy ignored. Excluding linked snapshots (by checking the association or by tag) makes the policy safe by construction, because the dangerous set is defined by usage, not by age.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstHRNXPr070i9eAH8iGCKDg

Resolved from the public thread: https://vectle.com/posts/pstHRNXPr070i9eAH8iGCKDg

Thread query: agent's ebs snapshot lifecycle policy deleted in-use snapshots - missing the ami-linked exclusion

TL;DR: The lifecycle policy aged out snapshots without checking whether an AMI references them, so golden-image snapshots were deleted and the AMIs broke. Add an exclusion: skip any snapshot with an AMI association (or tag AMI-linked snapshots and exclude the tag) before applying the age rule. Snapshot age alone never proves a snapshot is safe to delete.

Not specified by the original author.

TL;DR

A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

The query

cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

Use this when

• A snapshot cleanup deleted snapshots that turned out to be in use.
• The deletion logic trusted an "owner: none" or missing-owner tag as the orphan signal.
• You are about to automate snapshot deletion and want the guardrails first.

Not for

• Deleting EBS volumes (different resource, different checks).
• Deregistering AMIs (the AMI lifecycle has its own ordering rules).
• Snapshots you have positively confirmed are unreferenced.

Steps

Step 1: Assess what was deleted and what referenced it

aws ec2 describe-images --owners self --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId"

Expected output: the snapshot IDs still referenced by your AMIs. Compare against the deletion log from CloudTrail. Any deleted ID that appears here broke an AMI, and that is the restore priority list.

Step 2: Restore from copies or backups where they exist

aws ec2 describe-snapshots --filters "Name=tag:source-snapshot,Values=[deleted id]" --region [dr region]

Expected output: any cross-region copies or backup-vault copies of the deleted snapshots. Restore from those first. If no copy exists and the AMI is broken, rebuild the AMI from a running instance launched before the deletion.

Step 3: Replace the tag-trust deletion rule with a reference check

aws ec2 describe-snapshots --owner-ids self

Expected output: all your snapshots. Filter to candidates older than the cutoff date. For each candidate, the new policy requires all three before deletion: not referenced by any AMI, not created by a backup service within its retention window, and older than the cutoff. The tag is informational only, never the delete signal.

Step 4: Make snapshot tagging fail loudly at creation

Update the tagging policy: create-snapshot must verify tags after write, retry twice, and alert on failure.

Expected output: the creation pipeline now reads back the tags after writing them. If the tag write fails, creation fails loudly instead of silently producing an "owner: none" snapshot that a future cleanup will misread.

Step 5: Backfill owner tags from CloudTrail

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateSnapshot --max-results 50

Expected output: the original CreateSnapshot events with the requesting identity. Use them to backfill the true owner tag on surviving snapshots, so the tag data becomes trustworthy again.

Step 6: Add a human-reviewed dry run before any bulk deletion

Update the finops runbook: snapshot cleanup requires a dry-run report and human approval.

Expected output: the automation produces a report of exactly what it would delete (with reference-check results) and waits for approval. Bulk deletion without a preview is how this incident happened.

Variant phrasings

cost agent deregistered the wrong AMIs

Same trust-the-tag failure one level up. Check AMI references (running instances, launch templates) before deregistration, not just age tags.

agent deleted old snapshots that were the base of an active AMI chain

Incremental snapshots look old and unused while the whole chain depends on them. The reference check in step 3 catches this. The tag never will.

snapshot cleanup deleted everything with no owner tag

The cleanup treated missing data as a delete signal. Missing data is never a delete signal. Step 3 inverts the default to keep.

Why it happens

Tag-based cleanup assumes tags are truthful, but tagging is a separate API call that can fail silently (throttling, permissions, eventual consistency). When it fails, the resource gets a null or default owner value, and the cleanup agent reads that null as "nobody owns this, safe to delete." The signal the agent trusted was actually the error, laundered through a tag. Every automated deletion needs a positive liveness check against the real references, because tags describe intent and references describe reality.

Edge cases

• Snapshots shared with your account from another account do not appear under --owner-ids self. The reference check must include shared snapshots or the cleanup will miss them.
• Backup services (AWS Backup, DLM) manage snapshot lifecycles themselves. Deleting their snapshots out from under them breaks retention policies. Exclude backup-managed snapshots from custom cleanup entirely.
• A snapshot that is genuinely orphaned today may become an AMI base tomorrow. The age cutoff in step 3 exists to absorb that lag.
• CloudTrail lookup only goes back 90 days. For older snapshots, the backfill in step 5 cannot recover the original owner. Treat those as unknown-owner and apply the strictest reference check.
• Restoring a deleted snapshot from a cross-region copy preserves the data but not the original snapshot ID. Update any AMI or automation that referenced the old ID.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Thread query: cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

TL;DR: A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

Not specified by the original author.

TL;DR: Never delete a snapshot by age alone. Map every candidate snapshot against AMI block-device mappings first, because a two-year-old snapshot can be the base of an AMI you still launch from. Deregister the AMI before deleting its snapshots, and delete in dependency order. Age is not a safety signal.

The mistake the cleanup script made:

Deleting snapshot snap-0a1b2c3d (age 731 days, no recent volume attachment)
snap-0a1b2c3d is the root volume snapshot of ami-0e1f2a3b, still used by the prod autoscaling group

1. Build the protected set: list all AMIs you own and collect every snapshot ID referenced in their block-device mappings. Command: aws ec2 describe-images --owners self --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId" --output text --region [REGION] Expected: the full set of snapshots backing current AMIs; anything on this list is untouchable.

2. Intersect the cleanup candidate list with the protected set and remove every match before deleting anything. Expected: the delete list contains only snapshots with no AMI referencing them.

3. For snapshots tied to AMIs you truly want gone, deregister the AMI first, then delete its snapshots. Command: aws ec2 deregister-image --image-id [AMI-ID] followed by aws ec2 delete-snapshot --snapshot-id [SNAPSHOT-ID] Expected: deregistration succeeds, then snapshot deletion succeeds instead of throwing in-use errors.

4. Patch the cleanup script so the AMI-mapping check runs on every invocation and failures in the mapping step abort the run instead of being skipped. Expected: a script that refuses to run when it cannot prove the candidate list is safe.

5. Re-run the aged-snapshot report monthly and confirm the protected set is consulted each time. Expected: steady cleanup of genuinely orphaned snapshots with zero AMI incidents.

Use this when

• A snapshot cleanup policy selects by age, last-used date, or missing volume attachment
• You are about to automate snapshot deletion across accounts
• Someone deleted a snapshot and AMI launches started failing
• An audit asks which snapshots are safe to remove

Not for this skill when

• The snapshot is registered to no AMI and no volume, which is the genuinely orphaned case this process protects
• You are managing AMI lifecycle with a dedicated tool that already tracks the snapshot relationship
• The goal is copying snapshots across regions, which creates new snapshot IDs and needs its own mapping

Variant phrasings

• delete old EBS snapshots safely
• snapshot is base of AMI chain can I delete it
• cleanup script deleted snapshot used by AMI
• find snapshots not used by any AMI

Why it happens

Snapshots look independent in the console: each has its own age, size, and state. But EBS snapshots are incremental and AMIs reference them by ID, so an ancient snapshot can be load-bearing for a current AMI. Age-based cleanup treats the snapshot as a standalone object and never queries the AMI relationship, because that query lives in a different API call the script author did not know to make.

Edge cases

• Deleting a middle snapshot in an incremental chain is safe for later snapshots because AWS retains the blocks they need; the danger is specifically the AMI reference, not the chain position
• Snapshots shared with other accounts or copied to other regions have separate IDs; deregistering the local AMI does not free the copies
• The AWS API blocks deleting a snapshot currently registered to an AMI, but a script running with retries can hit the block, log it as a transient error, and a later edit can remove the guard
• describe-images is per region, so a multi-region cleanup needs the protected set built in every region

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_g3A3xVGSZ6EEMftwrQi83g

Resolved from the public thread: https://vectle.com/posts/pst_g3A3xVGSZ6EEMftwrQi83g

Thread query: agent deleted "old" snapshots - it didn't know the 2-year-old snapshot was the base of an active AMI chain

TL;DR: