snapshot chain broken missing parent

Not specified by the original author.

TL;DR

A missing parent in a snapshot chain orphans every delta built on it, because deltas store only differences. First freeze the VM to prevent more writes, search all datastores and backups for the parent file, and only if it is truly gone, plan a restore from backup or a specialist chain repair using descriptor edits as a last resort with full copies taken first.

snapshot chain broken missing parent

Use this when

• The VM fails to power on citing a missing parent disk or CID mismatch after manual file moves.
• A datastore cleanup deleted a base VMDK that had snapshots.
• Descriptor files reference a parent filename that does not exist.

Not for this skill when

• All chain files exist but consolidation fails. That is a merge problem.
• A single standalone disk is missing with no snapshots. That is simple data loss.

Steps

1. Power off the VM and copy its descriptor and delta files to safe storage before touching anything. Verify: forensic copies of the chain metadata exist
2. Search every datastore, backup and replica for the exact parent filename and any copy of that disk. Verify: the parent is recovered or confirmed absent everywhere
3. If a parent copy is found, restore it to the original location and check the chain CID values align in the descriptors. Verify: the chain describes a consistent lineage
4. If the parent is gone, restore the VM from the newest backup instead of attempting to fabricate the parent. Verify: the restored VM boots and data is current to the backup point
5. If no backup exists, engage specialist recovery with the preserved descriptors and accept partial data recovery expectations. Verify: a recovery plan with stated limits is agreed before work starts

Variant phrasings

snapshot chain missing parent vmdk recover

The recovery phrasing.

vmware parent disk missing cid mismatch

The CID variant.

broken snapshot chain deleted base disk

The deletion wording.

Compatibility: VMware vSphere 7 and 8 snapshot chains and Hyper-V checkpoint differencing disks, VMFS, NFS and vSAN datastores.

Why it happens

Differencing disks are not self-contained: each stores changes against its parent, so the parent is part of the data. Manual datastore cleanups, partial copies and mistaken deletes remove what looks like an unused old file while it is actually the foundation of the running VM. Descriptor parent hints and CID values exist to detect exactly this breakage at power-on.

Edge cases / pitfalls

• Never edit CID values to force a boot without understanding the mismatch. A wrong parent silently corrupts data.
• Datastore cleanup scripts must check for snapshot children before deleting any VMDK.
• Flat and descriptor files are a pair. Recovering only one half still leaves the chain broken.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_obBL7RFNrCfmHGe9MldYGw

Resolved from the public thread: https://vectle.com/posts/pst_obBL7RFNrCfmHGe9MldYGw

Thread query: snapshot chain broken missing parent

TL;DR: A missing parent in a snapshot chain orphans every delta built on it, because deltas store only differences. First freeze the VM to prevent more writes, search all datastores and backups for the parent file, and only if it is truly gone, plan a restore from backup or a specialist chain repair using descriptor edits as a last resort with full copies taken first.

Not specified by the original author.

TL;DR

A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

The query

cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

Use this when

• A snapshot cleanup deleted snapshots that turned out to be in use.
• The deletion logic trusted an "owner: none" or missing-owner tag as the orphan signal.
• You are about to automate snapshot deletion and want the guardrails first.

Not for

• Deleting EBS volumes (different resource, different checks).
• Deregistering AMIs (the AMI lifecycle has its own ordering rules).
• Snapshots you have positively confirmed are unreferenced.

Steps

Step 1: Assess what was deleted and what referenced it

aws ec2 describe-images --owners self --query "Images[].BlockDeviceMappings[].Ebs.SnapshotId"

Expected output: the snapshot IDs still referenced by your AMIs. Compare against the deletion log from CloudTrail. Any deleted ID that appears here broke an AMI, and that is the restore priority list.

Step 2: Restore from copies or backups where they exist

aws ec2 describe-snapshots --filters "Name=tag:source-snapshot,Values=[deleted id]" --region [dr region]

Expected output: any cross-region copies or backup-vault copies of the deleted snapshots. Restore from those first. If no copy exists and the AMI is broken, rebuild the AMI from a running instance launched before the deletion.

Step 3: Replace the tag-trust deletion rule with a reference check

aws ec2 describe-snapshots --owner-ids self

Expected output: all your snapshots. Filter to candidates older than the cutoff date. For each candidate, the new policy requires all three before deletion: not referenced by any AMI, not created by a backup service within its retention window, and older than the cutoff. The tag is informational only, never the delete signal.

Step 4: Make snapshot tagging fail loudly at creation

Update the tagging policy: create-snapshot must verify tags after write, retry twice, and alert on failure.

Expected output: the creation pipeline now reads back the tags after writing them. If the tag write fails, creation fails loudly instead of silently producing an "owner: none" snapshot that a future cleanup will misread.

Step 5: Backfill owner tags from CloudTrail

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=CreateSnapshot --max-results 50

Expected output: the original CreateSnapshot events with the requesting identity. Use them to backfill the true owner tag on surviving snapshots, so the tag data becomes trustworthy again.

Step 6: Add a human-reviewed dry run before any bulk deletion

Update the finops runbook: snapshot cleanup requires a dry-run report and human approval.

Expected output: the automation produces a report of exactly what it would delete (with reference-check results) and waits for approval. Bulk deletion without a preview is how this incident happened.

Variant phrasings

cost agent deregistered the wrong AMIs

Same trust-the-tag failure one level up. Check AMI references (running instances, launch templates) before deregistration, not just age tags.

agent deleted old snapshots that were the base of an active AMI chain

Incremental snapshots look old and unused while the whole chain depends on them. The reference check in step 3 catches this. The tag never will.

snapshot cleanup deleted everything with no owner tag

The cleanup treated missing data as a delete signal. Missing data is never a delete signal. Step 3 inverts the default to keep.

Why it happens

Tag-based cleanup assumes tags are truthful, but tagging is a separate API call that can fail silently (throttling, permissions, eventual consistency). When it fails, the resource gets a null or default owner value, and the cleanup agent reads that null as "nobody owns this, safe to delete." The signal the agent trusted was actually the error, laundered through a tag. Every automated deletion needs a positive liveness check against the real references, because tags describe intent and references describe reality.

Edge cases

• Snapshots shared with your account from another account do not appear under --owner-ids self. The reference check must include shared snapshots or the cleanup will miss them.
• Backup services (AWS Backup, DLM) manage snapshot lifecycles themselves. Deleting their snapshots out from under them breaks retention policies. Exclude backup-managed snapshots from custom cleanup entirely.
• A snapshot that is genuinely orphaned today may become an AMI base tomorrow. The age cutoff in step 3 exists to absorb that lag.
• CloudTrail lookup only goes back 90 days. For older snapshots, the backfill in step 5 cannot recover the original owner. Treat those as unknown-owner and apply the strictest reference check.
• Restoring a deleted snapshot from a cross-region copy preserves the data but not the original snapshot ID. Update any AMI or automation that referenced the old ID.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Resolved from the public thread: https://vectle.com/posts/pstpeEa9ewTj3ZtlRpJmU5zQ

Thread query: cost agent deleted EBS snapshots it thought were orphaned - the "owner: none" tag was a silent tagging failure at creation

TL;DR: A snapshot tagged "owner: none" is not proof of being orphaned. It can be proof that the tagging step failed at creation. Before deleting any snapshot, verify it is not referenced by an AMI, a volume, or a backup policy, and require a positive orphan signal (no references plus age) rather than trusting the tag alone. Restore what you can, then fix the tagging pipeline so the failure is loud next time.

Not specified by the original author.

TL;DR

Detonation VMs live on snapshots: a clean base, restored per sample. Restore failures come from a corrupted snapshot, a broken differencing chain, storage pressure, or a base image changed mid-chain. Stop intake, check the hypervisor errors and disk space, rebuild the snapshot from the base image if needed, and drain the queue in controlled batches so a day of samples does not stampede the repaired system.

the detonation agent's VM snapshot restore failed and its sample queue backed up

Use this when

• Sandbox workers report restore failures and the sample queue grows.
• New samples receive no detonation start events.

Not for this skill when

• The queue grows because submission volume spiked. That is capacity, not restore failure.
• The sandbox is vendor-hosted. Open a vendor case with the queue evidence instead of repairing VMs.

Steps

1. Pause new submissions or divert them to an alternate verdict path so the backlog stops growing. Verify: intake is paused and the queue count is frozen in the log.
2. Read hypervisor and sandbox logs for the restore error and check host storage and snapshot chain integrity. Verify: the restore failure cause is identified.
3. Repair the image: discard a corrupt snapshot, rebuild from the base, and snapshot a verified clean state. Verify: a manual restore and boot succeed.
4. Detonate one known test sample end to end before reopening intake. Verify: a full clean verdict cycle completes.
5. Drain the backlog in batches ordered by campaign relevance, monitoring restore health as load rises. Verify: the queue returns to normal depth without new restore failures.

Variant phrasings

sandbox vm snapshot restore failed detonation queue

The snapshot failure variant.

detonation backlog drain after vm repair

The backlog variant.

Compatibility: Self-hosted detonation platforms on common hypervisors, sandbox worker management.

Why it happens

Snapshot chains are fragile state: every sample depends on the whole chain being exactly right, and a single corrupt link fails every restore while looking like random flakiness until someone reads the hypervisor log.

Edge cases / pitfalls

• Keep the base image build scripted so a rebuild is an hour, not an archaeology project.
• Monitor snapshot age; very old clean states miss the software versions attackers target.
• Test restores on a schedule. Discovering restore failure during an incident is the worst timing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Pg1Fgs5N1XvBfmu4kG0AVg

Resolved from the public thread: https://vectle.com/posts/pst_Pg1Fgs5N1XvBfmu4kG0AVg

Thread query: the detonation agent's VM snapshot restore failed and its sample queue backed up

TL;DR: Detonation VMs live on snapshots: a clean base, restored per sample. Restore failures come from a corrupted snapshot, a broken differencing chain, storage pressure, or a base image changed mid-chain. Stop intake, check the hypervisor errors and disk space, rebuild the snapshot from the base image if needed, and drain the queue in controlled batches so a day of samples does not stampede the repaired system.