snapshot delta file growing indefinitely

Not specified by the original author.

TL;DR

A delta that grows indefinitely means snapshots are being created or left, but never consolidated, so every write lands in the delta forever. Find the snapshot owner and the failed consolidation behind it, merge during a planned window, and afterwards alert on snapshot age and delta size so the next stuck snapshot is caught in hours, not weeks.

snapshot delta file stuck growing indefinitely

Use this when

• One VM delta file is far larger or older than peers.
• Datastore consumption climbs daily with no new VMs.
• Snapshot manager shows an old snapshot nobody remembers creating.

Not for this skill when

• The delta belongs to an active backup right now. Growth during a job is normal.
• Storage array snapshots grow by design under a retention policy. Different system.

Steps

1. Identify the snapshot creation time, creator, and the VM current backing file to confirm it runs on the delta. Verify: snapshot owner and age are known
2. Look for failed consolidation or snapshot removal tasks and backup jobs tied to that snapshot. Verify: the reason it never merged is found
3. Confirm datastore headroom for the merge size and schedule consolidation in a low-activity window. Verify: space and window are secured
4. Consolidate or delete the snapshot through the hypervisor manager and monitor the merge to completion. Verify: the VM returns to its base disk
5. Add alerts for snapshots older than 48 hours and deltas over a size threshold. Verify: a test alert triggers on an artificial old snapshot report

Variant phrasings

delta disk growing snapshot not consolidating

The symptom phrasing.

vm running on snapshot delta huge

The size variant.

snapshot left behind delta growing fix

The leftover wording.

Compatibility: VMware vSphere 7 and 8 and Hyper-V checkpoint chains, standalone and backup-created snapshots.

Why it happens

A VM running on a delta writes every change there while the base disk stays frozen, so size grows with workload churn, not with data size. The condition persists because snapshot removal silently fails or a backup tool keeps a snapshot pinned, and no default alert screams about delta age. Performance also degrades as the chain lengthens and reads span more files.

Edge cases / pitfalls

• The longer a delta lives, the longer and riskier its merge. Early detection is the whole game.
• Reverting to an old forgotten snapshot destroys weeks of work. Consolidate forward, never revert, without owner sign-off.
• Some monitoring tools report VM disk size without showing delta breakdown. Inspect the datastore files directly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pkf46OPeREbmTKhBryZFKQ

Resolved from the public thread: https://vectle.com/posts/pst_pkf46OPeREbmTKhBryZFKQ

Thread query: snapshot delta file stuck growing indefinitely

TL;DR: A delta that grows indefinitely means snapshots are being created or left, but never consolidated, so every write lands in the delta forever. Find the snapshot owner and the failed consolidation behind it, merge during a planned window, and afterwards alert on snapshot age and delta size so the next stuck snapshot is caught in hours, not weeks.

Not specified by the original author.

TL;DR

Snapshot consolidation failed means delta disks could not merge back into the parent after backup. The usual blockers are a locked delta file, a VM still running on the delta, or too little datastore space for the merge. Consolidate from the hypervisor manager during low activity, never by deleting files by hand, and confirm the VM returns to its base disk.

veeam snapshot consolidation failed

Use this when

• Veeam or vSphere reports snapshot consolidation needed or failed.
• A VM has many delta files or one delta growing for days.
• Datastore space drops steadily while the VM appears healthy.

Not for this skill when

• Snapshots cannot be created in the first place. Fix creation first.
• The chain is broken because a parent file is missing entirely.

Steps

1. List the VM snapshot files and sizes in the datastore browser and note delta age and growth. Verify: the stuck deltas are identified by name and size
2. Check for locks by confirming no backup proxy or other VM has the disk mounted, and remove the disk from any proxy that still holds it. Verify: no other VM mounts the delta disk
3. Confirm free datastore space is at least the size of the largest delta plus headroom, freeing space or extending the datastore if not. Verify: space exceeds the merge requirement
4. Run consolidate snapshots from the hypervisor manager during a quiet period and monitor the task to completion. Verify: the task finishes and delta files disappear
5. Run the next Veeam job and confirm it creates and removes its snapshot cleanly. Verify: no new consolidation warning appears

Variant phrasings

vsphere snapshot consolidation failed locked

The locked-file phrasing.

veeam consolidation failed delta disk

The delta-disk variant.

vm snapshot consolidation needed fix

The vCenter warning wording.

Compatibility: VMware vSphere 7 and 8 with Veeam Backup and Replication 11 and 12, and Hyper-V checkpoint merging on Windows Server 2019 and 2022.

Why it happens

Consolidation rewrites every changed block from delta files into the parent disk. It needs exclusive access to the files and scratch space proportional to delta size. A proxy that kept the disk mounted, a backup that died mid-merge, or a nearly full datastore leaves the merge half done and every retry hits the same lock or space wall.

Edge cases / pitfalls

• A delta growing for weeks can take hours to merge. Plan the window instead of cancelling mid-merge.
• Cancelling a consolidation task rarely helps and can leave files in a worse state. Let it finish or fail on its own.
• Storage snapshots at the array level can also lock files. Check array jobs if hypervisor locks look clean.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstZfkXvE-G3Dsn-IimN2Uqw

Resolved from the public thread: https://vectle.com/posts/pstZfkXvE-G3Dsn-IimN2Uqw

Thread query: veeam snapshot consolidation failed

TL;DR: Snapshot consolidation failed means delta disks could not merge back into the parent after backup. The usual blockers are a locked delta file, a VM still running on the delta, or too little datastore space for the merge. Consolidate from the hypervisor manager during low activity, never by deleting files by hand, and confirm the VM returns to its base disk.

Not specified by the original author.

TL;DR

The snapshot contains malformed JSON (truncated writes, bad escaping, mixed encodings) and the diff's strict parser dies on it. Parse with error recovery, stream instead of loading everything, and quarantine the bad segments for inspection. A diff tool must survive ugly input.

Error

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Steps

1. Find the bad segment: binary-search the snapshot file for the offset where parsing fails. Expected: the exact corrupt region.
2. Decide: repair it (if the corruption pattern is known) or quarantine it and diff the rest. Expected: an explicit call, not a crash.
3. Switch the diff to streaming parse (one record at a time) instead of loading the whole snapshot. Expected: memory stays flat and one bad record cannot kill the run.
4. Wrap per-record parsing in error handling that logs and continues. Expected: the diff completes with a quarantine list.
5. Fix the snapshot writer so it cannot produce truncated JSON (atomic writes, checksums). Expected: the corruption stops at the source.

When to use

• Diff jobs crash with JSON parse errors.
• An agent snapshots CRM state and diffs it later.
• Large snapshots that may have been truncated.

When not to use

• Diffs that complete but look wrong (semantic issue).
• Schema changes between snapshots (different problem).

Tool compatibility

• Streaming JSON parsers; JSON lines format.
• Snapshot writers in the agent.

Variant phrasings

JSONDecodeError on snapshot

Find the bad segment; parse defensively.

diff crashed partway

Stream and quarantine.

Why it happens

Snapshots are written by one process and read much later by another. Anything that interrupts the write (crash, timeout, disk full) leaves truncated JSON that a strict parser rejects wholesale.

Edge cases

• JSON lines (one object per line) isolates corruption to single lines; prefer it for snapshots.
• Atomic writes (write temp, then rename) prevent torn files.
• Checksums on snapshot files catch corruption before the diff starts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstcxdWDYU2bovigJFzgpbwA

Resolved from the public thread: https://vectle.com/posts/pstcxdWDYU2bovigJFzgpbwA

Thread query: agent's snapshot diff crashed with json parse error

TL;DR: The snapshot contains malformed JSON (truncated writes, bad escaping, mixed encodings) and the diff's strict parser dies on it. Parse with error recovery, stream instead of loading everything, and quarantine the bad segments for inspection. A diff tool must survive ugly input.