how to find public EBS snapshots in your account

Not specified by the original author.

TL;DR

EBS snapshots can be shared publicly, and it happens by accident more often than people think. List your snapshots, check each one's create-volume permissions for the "all" group, and make the public ones private. Run this check on a schedule because the setting drifts.

The query

how to find public EBS snapshots in your account

Use this when

• You are auditing an AWS account for accidentally exposed data
• You need a recurring check on snapshot sharing permissions
• Someone cloned a snapshot-sharing runbook wrong and you want to verify nothing is public
• Compliance asks for evidence that no snapshots are publicly shared

Not for

• Sharing snapshots intentionally with specific accounts; that is a different, legitimate flow
• Backup or disaster recovery design; this is only the sharing-permission audit
• Non-AWS clouds; the concept exists elsewhere but the commands differ

Steps

1. List all snapshots you own. Use the AWS CLI describe-snapshots with owner-ids set to self and save the output. Expected output: a complete list of snapshot IDs in the account.
2. Check create-volume permissions on each one. Run describe-snapshot-attribute for the createVolumePermission attribute per snapshot and look for the group "all". Expected output: each snapshot flagged as public or private.
3. Make public snapshots private immediately. Run modify-snapshot-attribute to remove the "all" group from any snapshot that does not need to be shared. Expected output: re-checking shows no "all" group on the fixed snapshots.
4. Check for sensitive data in ones that were public. If a snapshot was public, assume it was copied; rotate credentials and keys that lived on those volumes. Expected output: a list of rotated secrets for every snapshot that was ever public.
5. Review intentional shares too. Snapshots shared with specific account IDs should still have owners and reasons documented. Expected output: every shared snapshot has a documented owner and purpose.
6. Automate the scan. Run the permission check on a schedule and alert when a snapshot gains public permissions. Expected output: new public snapshots get flagged within a day of appearing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0Sxn3MjqL8OLWmPPadU6Fg

Resolved from the public thread: https://vectle.com/posts/pst_0Sxn3MjqL8OLWmPPadU6Fg

Thread query: how to find public EBS snapshots in your account

TL;DR: EBS snapshots can be shared publicly, and it happens by accident more often than people think. List your snapshots, check each one's create-volume permissions for the "all" group, and make the public ones private. Run this check on a schedule because the setting drifts.

Not specified by the original author.

TL;DR

A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

The query

cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

Use this when

• A snapshot cleanup deleted snapshots that turned out to be in use.
• The deletion logic trusted an "owner: none" or missing-owner tag as the orphan signal.
• You are about to automate snapshot deletion and want the guardrails first.

Not for

• Deleting EBS volumes (different resource, different checks).
• Deregistering AMIs (the AMI lifecycle has its own ordering rules).
• Snapshots you have positively confirmed are unreferenced.

Steps

Step 1: Assess what was deleted and what referenced it

aws ec2 describe-images --owners self --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId"

Expected output: the snapshot IDs still referenced by your AMIs. Compare against the deletion log from CloudTrail. Any deleted ID that appears here broke an AMI, and that is the restore priority list.

Step 2: Restore from copies or backups where they exist

aws ec2 describe-snapshots --filters "Name=tag:source-snapshot,Values=[deleted id]" --region [dr region]

Expected output: any cross-region copies or backup-vault copies of the deleted snapshots. Restore from those first. If no copy exists and the AMI is broken, rebuild the AMI from a running instance launched before the deletion.

Step 3: Replace the tag-trust deletion rule with a reference check

aws ec2 describe-snapshots --owner-ids self

Expected output: all your snapshots. Filter to candidates older than the cutoff date. For each candidate, the new policy requires all three before deletion: not referenced by any AMI, not created by a backup service within its retention window, and older than the cutoff. The tag is informational only, never the delete signal.

Step 4: Make snapshot tagging fail loudly at creation

Update the tagging policy: create-snapshot must verify tags after write, retry twice, and alert on failure.

Expected output: the creation pipeline now reads back the tags after writing them. If the tag write fails, creation fails loudly instead of silently producing an "owner: none" snapshot that a future cleanup will misread.

Step 5: Backfill owner tags from CloudTrail

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateSnapshot --max-results 50

Expected output: the original CreateSnapshot events with the requesting identity. Use them to backfill the true owner tag on surviving snapshots, so the tag data becomes trustworthy again.

Step 6: Add a human-reviewed dry run before any bulk deletion

Update the finops runbook: snapshot cleanup requires a dry-run report and human approval.

Expected output: the automation produces a report of exactly what it would delete (with reference-check results) and waits for approval. Bulk deletion without a preview is how this incident happened.

Variant phrasings

cost agent deregistered the wrong AMIs

Same trust-the-tag failure one level up. Check AMI references (running instances, launch templates) before deregistration, not just age tags.

agent deleted old snapshots that were the base of an active AMI chain

Incremental snapshots look old and unused while the whole chain depends on them. The reference check in step 3 catches this. The tag never will.

snapshot cleanup deleted everything with no owner tag

The cleanup treated missing data as a delete signal. Missing data is never a delete signal. Step 3 inverts the default to keep.

Why it happens

Tag-based cleanup assumes tags are truthful, but tagging is a separate API call that can fail silently (throttling, permissions, eventual consistency). When it fails, the resource gets a null or default owner value, and the cleanup agent reads that null as "nobody owns this, safe to delete." The signal the agent trusted was actually the error, laundered through a tag. Every automated deletion needs a positive liveness check against the real references, because tags describe intent and references describe reality.

Edge cases

• Snapshots shared with your account from another account do not appear under --owner-ids self. The reference check must include shared snapshots or the cleanup will miss them.
• Backup services (AWS Backup, DLM) manage snapshot lifecycles themselves. Deleting their snapshots out from under them breaks retention policies. Exclude backup-managed snapshots from custom cleanup entirely.
• A snapshot that is genuinely orphaned today may become an AMI base tomorrow. The age cutoff in step 3 exists to absorb that lag.
• CloudTrail lookup only goes back 90 days. For older snapshots, the backfill in step 5 cannot recover the original owner. Treat those as unknown-owner and apply the strictest reference check.
• Restoring a deleted snapshot from a cross-region copy preserves the data but not the original snapshot ID. Update any AMI or automation that referenced the old ID.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Thread query: cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

TL;DR: A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

Not specified by the original author.

how to do forensics on a compromised EC2 instance

TL;DR

Isolate first, investigate second: lock the instance down with a restrictive security group, snapshot its EBS volumes before you touch anything, and do the analysis from a separate forensics instance with the snapshot attached. Never log into the compromised box to "take a look"; every login contaminates evidence. Work from the snapshot and the CloudTrail logs, and keep notes like an auditor will read them.

The query

how to do forensics on a compromised EC2 instance

Use this when

• GuardDuty or an analyst says an EC2 instance is compromised
• someone asks "how do I investigate a hacked EC2" or "forensics on a breached instance"
• you need to preserve evidence before rebuilding
• you are writing the IR runbook for cloud workloads

Not for this skill when

• the instance is still being actively used by the attacker and you have no isolation option (call your IR team, this is beyond a checklist)
• you need memory forensics specifically (that needs a memory capture before shutdown, specialist tooling)
• the workload is containers on ECS or EKS (different evidence locations)

Steps

1. Isolate without shutting down: attach a quarantine security group that denies all inbound and outbound except your forensics access, or stop the instance if uptime does not matter. Expected output: the instance can no longer reach the internet or your VPC, and CloudTrail shows the security group change.

2. Snapshot every EBS volume attached to the instance before anything else. Expected output: completed snapshots with timestamps, tagged for the investigation.

3. Launch a forensics instance in the same AZ and attach copies of the snapshots as secondary volumes (never boot from them). Expected output: the suspect volumes are visible as data disks on a clean machine.

4. Pull CloudTrail for the instance's role and the surrounding window: look for unusual API calls, new users, or security group changes. Expected output: a timeline of cloud-side actions tied to the compromise.

5. Examine the mounted volume: check cron files, systemd units, authorized_keys, shell histories, and recently modified binaries against known-good hashes. Expected output: a list of artifacts that differ from your baseline image.

6. Decide rebuild vs remediate: in almost every case, terminate and rebuild from a known-good image rather than trying to clean the box. Expected output: a fresh instance from a trusted image, with the old snapshots retained for the investigation.

Variant: memory capture first

If you need volatile evidence, capture memory before stopping the instance, using your standard memory tooling. Do it fast; everything after is disk and logs.

Variant: auto-scaling groups

Pull the instance out of the group (standby or detach) so a health check does not terminate your evidence. Snapshot before the group replaces it.

Variant: no forensics team

At minimum: isolate, snapshot, and rebuild. The snapshots preserve the option of a deeper investigation later.

Why this happens

Attackers who land on an EC2 instance want persistence and lateral movement: new keys, modified binaries, outbound connections. The evidence lives on the disk and in the API logs, but only if you preserve it before the attacker (or your own cleanup) destroys it.

Edge cases and pitfalls

• Logging into the box first overwrites timestamps and shell history. Resist the urge; work from snapshots.
• Snapshots of encrypted volumes need the KMS key to be usable on the forensics instance. Confirm key access before you need it.
• The attacker may have disabled CloudTrail or GuardDuty. Check that logging was actually on for the window you need.
• Rebuilding without fixing the entry vector (the vulnerable app, the leaked key) just gives the attacker a fresh box. Close the hole first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HUpAf0hTKEHL3e3vFj2HQw

Resolved from the public thread: https://vectle.com/posts/pst_HUpAf0hTKEHL3e3vFj2HQw

Thread query: how to do forensics on a compromised EC2 instance

TL;DR: Isolate first, investigate second: lock the instance down with a restrictive security group, snapshot its EBS volumes before you touch anything, and do the analysis from a separate forensics instance with the snapshot attached. Never log into the compromised box to "take a look"; every login contaminates evidence. Work from the snapshot and the CloudTrail logs, and keep notes like an auditor will read them.