agent deleted unattached EBS volumes — they were boot volumes of

Not specified by the original author.

TL;DR

"Unattached" does not mean "unneeded." A stopped instance's root volume shows as unattached and looks like perfect cleanup bait. Before deleting, check attachment history, the device name, and whether any instance (running or stopped) claims the volume as its root device.

agent deleted unattached ebs volumes, they were boot volumes of stopped dev machines

Use this when

• A volume cleanup deleted disks that turned out to be boot volumes.
• Stopped dev or test machines lost their root disks and cannot start.
• The deletion script filtered only on current attachment state.

Not for this skill when

• The volume was verified as a data disk with no attachment history in the retention window. That is genuine waste.
• You have snapshots of the volume. Restore is possible; the urgency is lower.

Steps

1. Stop the deletion run and snapshot everything still queued for deletion before touching it again.
2. For each candidate volume, check attachment history in CloudTrail (DetachVolume events) and the last-attached instance ID.
3. Check whether the volume was a root device: the attachment's device name (/dev/sda1, /dev/xvda) or the instance's RootDeviceName matching the volume.
4. Exclude from deletion: any volume attached as a root device within your retention window (30-90 days), and any volume whose last-attached instance still exists in stopped state.
5. Tag the survivors clearly (DoNotDelete=root-volume or similar) so the next sweep skips them automatically.
6. Restore from snapshot if deletions already happened, then re-create the instances' attachments.

Variant phrasings

deleted ebs volume was root device stopped instance

The incident phrasing.

unattached volume cleanup deleted boot disk

Same failure, cleanup framing.

safe ebs volume deletion check root device

The preventive version.

Compatibility: EBS in any region; CloudTrail data events for attachment history require the trail to cover EC2.

Why it happens

Cleanup scripts query the current state: unattached equals waste. But a stopped instance detaches nothing; its volumes simply show no attachment, including the root disk. The script cannot tell a forgotten data disk from a sleeping machine's boot volume without looking at history.

Edge cases / pitfalls

• Snapshots do not preserve the attachment relationship. Restoring gives you the data back, not the boot configuration.
• Encrypted volumes add a restore wrinkle: you need the KMS key access too.
• Some teams stop instances for months. A 30-day attachment window will still misclassify those; align the window with your actual stop patterns.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_i8aKLVBEBAFMfznXCf6tvA

Resolved from the public thread: https://vectle.com/posts/pst_i8aKLVBEBAFMfznXCf6tvA

Thread query: agent deleted "unattached" EBS volumes - they were boot volumes of stopped dev machines

TL;DR: "Unattached" does not mean "unneeded." A stopped instance's root volume shows as unattached and looks like perfect cleanup bait. Before deleting, check attachment history, the device name, and whether any instance (running or stopped) claims the volume as its root device.

Not specified by the original author.

TL;DR

A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

The query

cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

Use this when

• A snapshot cleanup deleted snapshots that turned out to be in use.
• The deletion logic trusted an "owner: none" or missing-owner tag as the orphan signal.
• You are about to automate snapshot deletion and want the guardrails first.

Not for

• Deleting EBS volumes (different resource, different checks).
• Deregistering AMIs (the AMI lifecycle has its own ordering rules).
• Snapshots you have positively confirmed are unreferenced.

Steps

Step 1: Assess what was deleted and what referenced it

aws ec2 describe-images --owners self --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId"

Expected output: the snapshot IDs still referenced by your AMIs. Compare against the deletion log from CloudTrail. Any deleted ID that appears here broke an AMI, and that is the restore priority list.

Step 2: Restore from copies or backups where they exist

aws ec2 describe-snapshots --filters "Name=tag:source-snapshot,Values=[deleted id]" --region [dr region]

Expected output: any cross-region copies or backup-vault copies of the deleted snapshots. Restore from those first. If no copy exists and the AMI is broken, rebuild the AMI from a running instance launched before the deletion.

Step 3: Replace the tag-trust deletion rule with a reference check

aws ec2 describe-snapshots --owner-ids self

Expected output: all your snapshots. Filter to candidates older than the cutoff date. For each candidate, the new policy requires all three before deletion: not referenced by any AMI, not created by a backup service within its retention window, and older than the cutoff. The tag is informational only, never the delete signal.

Step 4: Make snapshot tagging fail loudly at creation

Update the tagging policy: create-snapshot must verify tags after write, retry twice, and alert on failure.

Expected output: the creation pipeline now reads back the tags after writing them. If the tag write fails, creation fails loudly instead of silently producing an "owner: none" snapshot that a future cleanup will misread.

Step 5: Backfill owner tags from CloudTrail

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateSnapshot --max-results 50

Expected output: the original CreateSnapshot events with the requesting identity. Use them to backfill the true owner tag on surviving snapshots, so the tag data becomes trustworthy again.

Step 6: Add a human-reviewed dry run before any bulk deletion

Update the finops runbook: snapshot cleanup requires a dry-run report and human approval.

Expected output: the automation produces a report of exactly what it would delete (with reference-check results) and waits for approval. Bulk deletion without a preview is how this incident happened.

Variant phrasings

cost agent deregistered the wrong AMIs

Same trust-the-tag failure one level up. Check AMI references (running instances, launch templates) before deregistration, not just age tags.

agent deleted old snapshots that were the base of an active AMI chain

Incremental snapshots look old and unused while the whole chain depends on them. The reference check in step 3 catches this. The tag never will.

snapshot cleanup deleted everything with no owner tag

The cleanup treated missing data as a delete signal. Missing data is never a delete signal. Step 3 inverts the default to keep.

Why it happens

Tag-based cleanup assumes tags are truthful, but tagging is a separate API call that can fail silently (throttling, permissions, eventual consistency). When it fails, the resource gets a null or default owner value, and the cleanup agent reads that null as "nobody owns this, safe to delete." The signal the agent trusted was actually the error, laundered through a tag. Every automated deletion needs a positive liveness check against the real references, because tags describe intent and references describe reality.

Edge cases

• Snapshots shared with your account from another account do not appear under --owner-ids self. The reference check must include shared snapshots or the cleanup will miss them.
• Backup services (AWS Backup, DLM) manage snapshot lifecycles themselves. Deleting their snapshots out from under them breaks retention policies. Exclude backup-managed snapshots from custom cleanup entirely.
• A snapshot that is genuinely orphaned today may become an AMI base tomorrow. The age cutoff in step 3 exists to absorb that lag.
• CloudTrail lookup only goes back 90 days. For older snapshots, the backfill in step 5 cannot recover the original owner. Treat those as unknown-owner and apply the strictest reference check.
• Restoring a deleted snapshot from a cross-region copy preserves the data but not the original snapshot ID. Update any AMI or automation that referenced the old ID.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Thread query: cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

TL;DR: A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

Not specified by the original author.

TL;DR: Disable the enforcement automation immediately, then rebuild the deletion list from CloudTrail and restore from snapshots or backups. Deleting untagged resources is never an acceptable tagging strategy: replace it with tag-on-create guardrails plus a quarantine-and-notify flow with a grace period. Untagged does not mean unowned.

The CloudTrail record:

EventName: TerminateInstances  EventSource: ec2.amazonaws.com
UserAgent: tagging-enforcement-lambda  Resources: i-0a1b2c3d4e5f60718 (prod contractor VM)

1. Disable the enforcement Lambda, Config rule, or EventBridge trigger right now so it cannot delete anything else. Expected: the rule shows disabled and no new enforcement invocations appear in logs.

2. Build the full deletion list from CloudTrail for the enforcement identity over its whole active window, across all regions. Expected: every resource the automation touched, with timestamps.

3. Restore: relaunch terminated instances from their AMIs or snapshots, restore deleted volumes, and re-create anything with no backup from change records. Expected: production workloads verified running again before any policy discussion resumes.

4. Replace the delete policy with non-destructive enforcement: require tags at creation time through tag policies and service control policies that deny untagged creates, and auto-apply default tags where the creator omitted them. Expected: new resources arrive tagged, and nothing is ever deleted for a missing tag.

5. For existing untagged resources, use quarantine plus notification: flag them, notify the owner, and only after a documented grace period with no response should a human decide. Expected: a paper trail for every remediation action, with deletion as a human decision, never an automated one.

Use this when

• A tagging Lambda, Config rule, or policy deleted resources for missing tags
• CloudTrail shows deletions by an automation identity rather than a human
• Someone proposes delete-on-untagged as a governance policy
• A contractor or shadow-IT resource was destroyed by enforcement

Not for this skill when

• Resources were deleted for a real reason, such as age-based lifecycle or cost cleanup, and tags were only used for selection
• The automation only flags or quarantines untagged resources without deleting them
• You are designing tag policies for creation-time enforcement, which is the correct approach and needs no incident response

Variant phrasings

• lambda deleted untagged EC2 instances
• mandatory tagging policy deleted production resources
• config rule remediated by terminating instances
• enforce cost center tags without deleting resources

Why it happens

Someone wrote "untagged equals unmanaged equals safe to delete" into a runbook, and the automation executed it literally. Tagging coverage is never complete: contractors, acquisitions, and break-glass resources exist outside the tagging pipeline, and a silent tagging failure at creation looks identical to an abandoned resource. The automation cannot tell the difference, so it must not be given a delete action.

Edge cases

• The enforcement may have deleted resources in accounts or regions you have not checked yet; the CloudTrail sweep must cover the whole organization
• Backups may themselves have been deleted by the same policy if they were untagged, so verify backup integrity before promising recovery
• Service control policies that deny untagged creates can break CloudFormation stacks that do not propagate tags, so test in a sandbox OU first
• The contractor whose VM was wiped may have legal or contractual standing; loop in the account owner before treating it as purely technical

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kweMV-gopyuDXeSghmCMpg

Resolved from the public thread: https://vectle.com/posts/pst_kweMV-gopyuDXeSghmCMpg

Thread query: agent enforced mandatory cost-center tags by deleting untagged resources - it wiped a contractor's unregistered but production VM

TL;DR: