agent reported disk space preflight passed using stale metrics and install failed

TL;DR

Preflight is only as good as its timestamp. If the agent checks disk space hours or days before the install, or reuses inventory metrics, a host can pass with space it no longer has. Make disk checks just-in-time: measure free space on the relevant filesystems at dispatch, again at install start, compare against a requirement that includes download, extraction, and rollback headroom, and fail the host fast with fresh numbers when it falls short. Log the measured values with their timestamps in the host record.

agent reported disk space preflight passed using stale metrics and install failed

Use this when

• Installs fail with out of space errors on hosts whose preflight records show comfortable free space.
• Preflight data is collected in a nightly inventory or at wave planning time, well before installs run.
• The failing hosts have active workloads that consume disk between planning and patching, such as logs, queues, or user data.

Not for this skill when

• Preflight was measured minutes before install and the install itself generated more data than the host had. That is a requirement sizing problem, not staleness.
• The wrong filesystem was measured - for example root checked while packages install to a separate volume. Fix the measurement target.

Steps

1. Prove the staleness on one failed host: compare the preflight recorded free space and its timestamp with the host actual free space history - from monitoring if available - at install time and now. Quantify how much space disappeared between the check and the install, and what consumed it if the host logs make that obvious. Verify: you have timestamped numbers showing the gap between preflight space and install time space for a real failure
2. Set a real space requirement per update class: measure what a cumulative or kernel update actually needs on each platform - download size, extracted working size, and safe headroom - rather than a single guessed threshold. Different filesystems matter too: root, boot, var, and the package cache location each need their own check where they are separate mounts. Verify: documented per-class requirements exist for each relevant filesystem, based on measured installs
3. Move the check to dispatch time: the agent measures free space on the target host immediately before sending the install, using a live query to the host, and aborts that host with a disk blocked state and the fresh figures if it falls below requirement. The wave planning estimate can stay as a forecast, but it must never be the gate. Verify: a test host whose disk is filled after planning is blocked at dispatch with fresh numbers in its record
4. Add a second check at install start on the host side where possible, because even minutes matter on fast filling disks, and have the install wrapper report the measured space back so the tracker holds the number that actually governed the install. If space fell below the floor between dispatch and start, the wrapper stops before the package manager does any work. Verify: the host record shows the install start measurement, and a filled host stops cleanly before its package manager runs
5. Watch the trend, not just the gate: report disk blocked hosts per wave with their shortfall, and feed chronic offenders to a disk remediation process - log rotation, old kernel cleanup, cache limits - so the same hosts do not block every wave. A rising blocked count across waves is a fleet disk health signal the patch team should surface, not silently absorb. Verify: each wave report lists disk blocked hosts with shortfalls, and repeat offenders have an open remediation action

Why it happens

Disk space is a fast-moving measurement on busy systems: logs rotate, queues drain and fill, users and applications write data continuously. An agent architecture that separates planning from execution by hours or days inherits whatever drift happens in between, and package installs are unforgiving - they need their full working space at the worst moment, mid transaction. The stale metric failure feels like an agent logic bug but it is really a data freshness bug: the decision was correct for the moment it measured, and wrong for the moment it mattered. Just-in-time measurement at the point of commitment, with requirements sized from real installs, is the standard fix for any preflight that gates a state-changing action.

Variants and edge cases

Metrics come from a central inventory the agent cannot query live

Add a lightweight live check for the gating values only, and keep inventory data for planning. If live checks are impossible for a host class, shorten the allowed metric age and enforce it.

Inodes exhaust while bytes remain

Measure inode availability as well as bytes on filesystems that hold many small files, such as mail or cache volumes, or the fresh check will pass a host that still cannot install.

Space is consumed by the patch download itself on small hosts

Stage downloads to a larger filesystem where the platform supports it, or sequence download and install in smaller batches for constrained host classes.

Compatibility: Applies to patch agents for Windows and Linux fleets using live host queries through WinRM, SSH, or management agents such as Configuration Manager.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mPgQg2bqBcCl1g3bBdz9ug