agent deployed linux kernel update but never regenerated initramfs on target hosts

Not specified by the original author.

TL;DR

Installing kernel files is only half the job: the new kernel needs a matching initramfs built for that host storage and drivers, or the first reboot can fail or fall back. Make the agent verify a fresh initramfs exists for the exact new kernel version - correct name, plausible size, timestamp after the kernel install - and regenerate it with the distribution tool when it does not. Hold the reboot until that check passes, and keep the previous kernel as the boot fallback.

agent deployed linux kernel update but never regenerated initramfs on target hosts

Use this when

• Hosts reboot after a kernel update and fail, fall back to the old kernel, or lose storage or network devices the old kernel had.
• The boot folder shows the new kernel image but its initramfs is missing, tiny, or older than the kernel.
• The agent disables package scripts, uses a custom install path, or copies kernels instead of installing through the package manager.

Not for this skill when

• The initramfs is present and current, and the new kernel fails for a driver bug. That is a kernel problem, not a generation problem.
• The host uses a unified kernel image or a different boot architecture by design. Follow that design verification instead.

Steps

1. Inventory the damage on affected hosts before rebooting any more: for each target host, list the boot folder and compare every installed kernel version against its initramfs partner by name and timestamp. Flag every mismatch, because hosts still running are one reboot away from the failure and should be fixed before that reboot happens by accident or by schedule. Verify: every host is classified as matched, missing initramfs, or stale initramfs, from direct boot folder evidence
2. Regenerate the missing images with the distribution native tool for the exact kernel versions: dracut on Red Hat family systems or update-initramfs on Debian and Ubuntu, watching the output for missing driver or firmware warnings, especially for the storage controller the host actually boots from. A generation that completes while omitting the boot storage driver produces an image that still cannot boot the root filesystem. Verify: each regenerated initramfs completes without unresolved storage driver warnings and its size is in line with the previous kernel image
3. Verify the contents, not just the file: list the modules inside the new initramfs or check the generation log to confirm the storage, filesystem, and if relevant LVM or encryption drivers for this host are included. Hosts with unusual storage need their host-specific generation configuration respected, which generic images built elsewhere will not have. Verify: the new initramfs demonstrably contains the drivers this host needs to mount its root filesystem
4. Fix the agent step so generation is part of installation: after any kernel package change, the agent runs or confirms the package own generation hooks, then performs the name, size, and timestamp verification as a gate. If the gate fails, the agent regenerates once, and if it still fails, marks the host blocked and does not schedule its reboot. Verify: a test kernel install passes the gate with a fresh verified initramfs, and a host with generation sabotaged in testing is blocked before reboot
5. Reboot one repaired host as the proof, in a window, and confirm it comes up on the new kernel with root filesystem and devices healthy. Only then release the remaining repaired hosts, and record initramfs verification as a standard line in the wave report so its absence is visible in future waves. Verify: the proof host boots the new kernel cleanly and the wave report shows initramfs verification per host

Why it happens

The initramfs is the temporary root filesystem the kernel uses to load the drivers needed to reach the real root filesystem. It is generated per host because it embeds that host storage, encryption, and filesystem setup. Package managers normally generate it automatically in the kernel install scripts, so it goes missing when an agent bypasses those scripts, interrupts them, copies kernel files manually, or when generation fails silently on a full boot partition. The kernel package then reports installed while the boot chain is incomplete - a trap that only springs at the next reboot, often days later and detached from the patch window that caused it. Verifying the image at install time keeps the cause and the effect in the same window.

Variants and edge cases

Boot partition too small for another initramfs

Generation fails or truncates when the boot filesystem fills with old kernels. Remove old kernels through the package manager, keeping the running and one fallback kernel, then regenerate.

Agent builds one golden initramfs for all hosts

Host-specific storage makes golden images risky. Generate on each host, or restrict shared images to provably identical hardware and storage layouts.

New kernel boots but without network or storage devices

A minimal initramfs or a fallback image was used. Regenerate in the full host mode rather than host-only minimal mode if your distribution distinguishes them, and include the missing drivers explicitly.

Compatibility: Applies to Linux hosts on Red Hat family with dracut and Debian or Ubuntu with initramfs-tools, on physical and virtual machines using GRUB.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstZdTcyrP9OD70aybqm3hUg

Resolved from the public thread: https://vectle.com/posts/pstZdTcyrP9OD70aybqm3hUg

Thread query: agent deployed linux kernel update but never regenerated initramfs on target hosts

TL;DR: Installing kernel files is only half the job: the new kernel needs a matching initramfs built for that host storage and drivers, or the first reboot can fail or fall back. Make the agent verify a fresh initramfs exists for the exact new kernel version - correct name, plausible size, timestamp after the kernel install - and regenerate it with the distribution tool when it does not. Hold the reboot until that check passes, and keep the previous kernel as the boot fallback.