agent's restore sandbox ran out of inodes during extraction and verification failed

TL;DR

Free disk blocks are not the problem, inodes are, so measure and budget them before extraction starts. Count the files in the backup catalog, add at least thirty percent headroom, check free inodes on the sandbox filesystem, and either pick a filesystem that grows inodes dynamically or pre-create the extraction target with enough inodes. Then extract in catalog segments with a checkpoint after each segment, monitor inode use as a watermark during the run, and abort early with a clear inode verdict instead of corrupting a half-finished verification.

agent's restore sandbox ran out of inodes during extraction and verification failed

Use this when

• Extraction fails with no space left messages while the disk still shows gigabytes free.
• The restored tree holds very many small files, mail spools, node modules, or container layer content.
• Verification fails at a different file each run, always deep into the extraction.

Not for this skill when

• The sandbox is out of bytes rather than inodes, shown by both block and inode counts being exhausted.
• A single archive is corrupt and fails its own integrity check before extraction.

Steps

1. Preflight both resources, not one. Query the sandbox filesystem for free blocks and free inodes and log both values in the run manifest, because most harness templates only check disk size and therefore walk straight into this failure. Compare free inodes against the catalog file count plus directory count, multiplied by a headroom factor of at least 1.3 to cover temporary files the extractor and the checker create. Verify: the manifest records free inodes, required inodes, and a pass or fail preflight verdict
2. Choose or build a target that can hold the inode count. Prefer a filesystem with dynamic inode allocation for the sandbox, or create the image with an explicit inode count sized from the estimate when using a fixed inode filesystem, and record which choice was made. Keep the extraction target separate from the sandbox root filesystem so a full extraction cannot starve the agent runtime, its logs, or its temporary directory. Verify: a test extraction of the largest known small-file tree completes without the inode watermark passing eighty percent
3. Extract in segments with checkpoints. Split the restore by top level directory or by catalog segment, extract one segment, record its file count and cumulative inode use in a checkpoint file, and only then continue. If the run dies, resume from the last checkpoint and skip finished segments instead of restarting into the same wall. Never extract the same segment twice into the same tree, because duplicate partial files consume inodes without adding evidence. Verify: killing the extractor mid-run and resuming finishes the tree with no segment extracted twice
4. Monitor inodes as a live watermark. Poll inode use every thirty seconds during extraction and during the verification scan, and emit progress events with files extracted, inodes used, and inodes remaining. When remaining inodes fall below ten percent of the estimate, pause and raise a specific inode warning rather than letting tools fail one by one with misleading errors. Verify: the run log shows inode readings from start to finish, not only a final failure
5. Account for the checker itself. Verification tools create lock files, hash caches, temporary extraction copies, and sometimes a second unpacked copy for comparison, all of which consume inodes on the same filesystem. Budget those explicitly, place agent temporary files on a different filesystem, and point the checker cache at the roomier target. Verify: the preflight budget lists extractor, checker, and agent temporary inode needs as separate lines
6. Report the real cause and resize the sandbox profile. When this failure fires, record it as inode exhaustion with the counts, not as a generic extraction error, and update the sandbox size profile for this backup job so the next run allocates the larger inode budget automatically. Retire stale sandbox images whose fixed inode count was set for a much smaller tree. Verify: the next scheduled run of the same job passes preflight without a manual override

Why it happens

Many small files consume one inode each while using almost no blocks, so a sandbox sized by gigabytes can be inode-full at a few percent disk use. Agent harnesses inherit that blind spot from disk space checks that only look at bytes, and extraction tools report the failure as a generic write error, which sends the agent toward corrupt archive theories. The catalog already knows the file count, so the fix is to turn that count into an explicit budget before any bytes move.

Variants and edge cases

Edge cases: hard links in the source may extract as separate files on filesystems or tools that do not preserve links, inflating the inode need well past the catalog count, so sample-extract a link-heavy directory when sizing. Overlay filesystems report the upper layer inode pool, which may be shared with other sandboxes on the host. A shared sandbox host can therefore fail one run because a neighbour consumed the pool, which is why the manifest should record host level inode use too.

sandbox no space left but disk is empty during restore

Use when the error text says no space while block use is low. Compare free inodes against the catalog count first.

extraction failed on thousands of small files

Use when the tree is small-file heavy. Segment the extraction and keep checkpoints so a resize does not restart the whole job.

restore verify ran out of inodes on tmpfs

Use when the sandbox uses a memory filesystem with a default inode cap. Raise the inode count explicitly at mount time or move the target.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstiSOhmB4OlF6lykXuMxZVg