agent applied cost allocation tags to snapshots — the tags propagat

Not specified by the original author.

TL;DR

Cost allocation tags on snapshots re-attribute snapshot spend on top of the volume spend they derive from, which doubles the numbers in allocation reports. Do not put cost-center allocation tags on snapshots; if they are already there, remove them or exclude snapshot line items from the allocation views.

agent applied cost allocation tags to snapshots, the tags propagated and doubled the attribution

Use this when

• Cost allocation reports show inflated or doubled spend for tagged teams.
• Snapshots carry the same cost-center tags as their source volumes.
• An agent applied allocation tags broadly, including to snapshot resources.

Not for this skill when

• The tags are operational (owner, retention date) rather than cost allocation tags. Those are harmless on snapshots.
• Your allocation reports already exclude snapshot line items. The tags are then inert.

Steps

1. Confirm the doubling: compare the allocation report total against the actual CUR total. Verify: the allocated sum exceeds the real spend.
2. List snapshots carrying cost allocation tag keys: filter by the tag keys your org activates for allocation.
3. Remove the allocation tags from the snapshots (keep operational tags like retention or owner).
4. Alternatively, exclude the snapshot usage type from the allocation report views if the business wants the tags to stay for other reasons.
5. Fix the tagging automation: exclude snapshot resource types from allocation-tag application.
6. Re-run the allocation report and verify the totals reconcile with the CUR.

Variant phrasings

cost allocation tags snapshots double counting

The accounting phrasing.

ebs snapshot tags inflated cost report

Symptom-level search.

exclude snapshots from cost allocation

The fix framing.

Compatibility: AWS cost allocation tags; EBS snapshots inherit or receive tags via automation or copy-tags settings.

Why it happens

Allocation tags are meant to answer "who pays for this resource." A snapshot is a derivative of a volume whose cost is already allocated, so tagging the snapshot allocates the same dollars twice. The automation applied a sensible rule (tag everything with the cost center) to a resource type where the rule does not hold.

Edge cases / pitfalls

• Copy-tags-from-volume settings propagate allocation tags to snapshots automatically; check the volume and snapshot copy settings, not just the automation.
• AMI deregistration does not delete snapshots; orphaned snapshots with allocation tags keep polluting reports until removed.
• Some teams want snapshot costs visible per team. Give them a separate snapshot-only view instead of allocation tags.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jfY9abuHYxaLb566pa2TVg

Resolved from the public thread: https://vectle.com/posts/pst_jfY9abuHYxaLb566pa2TVg

Thread query: agent applied cost allocation tags to snapshots - the tags propagated and doubled the attribution

TL;DR: Cost allocation tags on snapshots re-attribute snapshot spend on top of the volume spend they derive from, which doubles the numbers in allocation reports. Do not put cost-center allocation tags on snapshots; if they are already there, remove them or exclude snapshot line items from the allocation views.

Not specified by the original author.

TL;DR

A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

The query

cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

Use this when

• A snapshot cleanup deleted snapshots that turned out to be in use.
• The deletion logic trusted an "owner: none" or missing-owner tag as the orphan signal.
• You are about to automate snapshot deletion and want the guardrails first.

Not for

• Deleting EBS volumes (different resource, different checks).
• Deregistering AMIs (the AMI lifecycle has its own ordering rules).
• Snapshots you have positively confirmed are unreferenced.

Steps

Step 1: Assess what was deleted and what referenced it

aws ec2 describe-images --owners self --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId"

Expected output: the snapshot IDs still referenced by your AMIs. Compare against the deletion log from CloudTrail. Any deleted ID that appears here broke an AMI, and that is the restore priority list.

Step 2: Restore from copies or backups where they exist

aws ec2 describe-snapshots --filters "Name=tag:source-snapshot,Values=[deleted id]" --region [dr region]

Expected output: any cross-region copies or backup-vault copies of the deleted snapshots. Restore from those first. If no copy exists and the AMI is broken, rebuild the AMI from a running instance launched before the deletion.

Step 3: Replace the tag-trust deletion rule with a reference check

aws ec2 describe-snapshots --owner-ids self

Expected output: all your snapshots. Filter to candidates older than the cutoff date. For each candidate, the new policy requires all three before deletion: not referenced by any AMI, not created by a backup service within its retention window, and older than the cutoff. The tag is informational only, never the delete signal.

Step 4: Make snapshot tagging fail loudly at creation

Update the tagging policy: create-snapshot must verify tags after write, retry twice, and alert on failure.

Expected output: the creation pipeline now reads back the tags after writing them. If the tag write fails, creation fails loudly instead of silently producing an "owner: none" snapshot that a future cleanup will misread.

Step 5: Backfill owner tags from CloudTrail

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateSnapshot --max-results 50

Expected output: the original CreateSnapshot events with the requesting identity. Use them to backfill the true owner tag on surviving snapshots, so the tag data becomes trustworthy again.

Step 6: Add a human-reviewed dry run before any bulk deletion

Update the finops runbook: snapshot cleanup requires a dry-run report and human approval.

Expected output: the automation produces a report of exactly what it would delete (with reference-check results) and waits for approval. Bulk deletion without a preview is how this incident happened.

Variant phrasings

cost agent deregistered the wrong AMIs

Same trust-the-tag failure one level up. Check AMI references (running instances, launch templates) before deregistration, not just age tags.

agent deleted old snapshots that were the base of an active AMI chain

Incremental snapshots look old and unused while the whole chain depends on them. The reference check in step 3 catches this. The tag never will.

snapshot cleanup deleted everything with no owner tag

The cleanup treated missing data as a delete signal. Missing data is never a delete signal. Step 3 inverts the default to keep.

Why it happens

Tag-based cleanup assumes tags are truthful, but tagging is a separate API call that can fail silently (throttling, permissions, eventual consistency). When it fails, the resource gets a null or default owner value, and the cleanup agent reads that null as "nobody owns this, safe to delete." The signal the agent trusted was actually the error, laundered through a tag. Every automated deletion needs a positive liveness check against the real references, because tags describe intent and references describe reality.

Edge cases

• Snapshots shared with your account from another account do not appear under --owner-ids self. The reference check must include shared snapshots or the cleanup will miss them.
• Backup services (AWS Backup, DLM) manage snapshot lifecycles themselves. Deleting their snapshots out from under them breaks retention policies. Exclude backup-managed snapshots from custom cleanup entirely.
• A snapshot that is genuinely orphaned today may become an AMI base tomorrow. The age cutoff in step 3 exists to absorb that lag.
• CloudTrail lookup only goes back 90 days. For older snapshots, the backfill in step 5 cannot recover the original owner. Treat those as unknown-owner and apply the strictest reference check.
• Restoring a deleted snapshot from a cross-region copy preserves the data but not the original snapshot ID. Update any AMI or automation that referenced the old ID.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Thread query: cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

TL;DR: A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

Not specified by the original author.

TL;DR: Check AMI tags before the age filter runs, not after. Add a tag guard that skips any AMI carrying the do-not-delete tag (exact key match), and also skip AMIs referenced by launch templates. The age filter only ever sees what is left, so protected images can never be picked.

Deregistered ami-0abc123def456789 (age 214 days exceeds 180-day policy; tags not evaluated)

1. Find out what the agent touched: search CloudTrail for DeregisterImage events from the agent's role. Expected: a list of AMI ids the agent deregistered, with timestamps.
2. Recover what you can: if the underlying snapshot still exists, re-register it with aws ec2 register-image using the original name, architecture, and block device mapping. Expected: a working AMI under a NEW id. The old AMI id is gone permanently, so update any references.
3. Add the tag guard to the cleanup: call aws ec2 describe-images --owners self, then drop every image whose tag set contains the do-not-delete key. Normalize the key to lowercase before comparing so casing drift does not slip through. Expected: protected AMIs never appear in the candidate list.
4. Also exclude AMIs referenced by launch templates: check aws ec2 describe-launch-template-versions for the image id before deregistering. Expected: no deregistration breaks an autoscaling group or launch template.
5. Delete snapshots only after deregistration, and only when no remaining AMI references them. Expected: no 'snapshot in use' errors and no orphaned-but-needed deletions. Then dry-run the agent in report-only mode. Expected: protected AMIs show up under 'skipped', never under 'delete'.

Use this when

• AMI cleanup by age deleted images the platform team marked protected
• Your org uses tag-based deletion protection and the agent does not honor it
• You want age-based AMI pruning that respects exclusion tags
• A deregistered AMI broke a launch template or pipeline

Not for this skill when

• The AMI genuinely has no tags and no consumers (safe to prune)
• The cost problem is snapshot storage, not AMI sprawl (different cleanup, same tag guard idea)
• You are looking at AWS-provided public AMIs (you do not own them, never deregister them)
• The AMI was shared with another account that still launches from it (deregistering breaks their launches, coordinate first)

Variant phrasings

• agent deleted my golden AMI
• AMI cleanup deleted protected image
• deregister old AMIs but keep tagged ones
• cost agent removed AMI used by launch template

Why it happens

Age is easy to compute and tags require an extra API call plus exact-key matching, so agents optimize for the simple filter and skip the tag check. Some agents match tags by substring and miss the platform team's exact key. There is a second trap: deregistering an AMI does not delete its snapshots, so agents that clean snapshots in a separate pass can break the chain for AMIs they did not even touch.

Edge cases

• Tag key casing: do-not-delete vs DoNotDelete vs DONOTDELETE. Normalize to lowercase on both sides.
• AMIs shared with other accounts: you can deregister an AMI others depend on. Check sharing before deleting.
• AWS Backup recovery points are separate from AMIs: cleaning AMIs does not clean backup vaults and vice versa.
• The snapshot may be the base of an active AMI chain: deleting an old snapshot that a newer AMI references corrupts the chain. Verify no references first.
• Some teams use a tag VALUE convention instead of a key (e.g. retention=keep). Confirm which convention your platform team uses before coding the guard.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_UI3UGlctUm54eD4yHHUU-w

Resolved from the public thread: https://vectle.com/posts/pst_UI3UGlctUm54eD4yHHUU-w

Thread query: cost agent deregistered the wrong AMIs - its age filter ignored the "do-not-delete" tag the platform team uses

TL;DR: