agent deployed linux kernel update but never regenerated initramfs on ta

Not specified by the original author.

TL;DR

Installing kernel files is only half the job: the new kernel needs a matching initramfs built for that host storage and drivers, or the first reboot can fail or fall back. Make the agent verify a fresh initramfs exists for the exact new kernel version - correct name, plausible size, timestamp after the kernel install - and regenerate it with the distribution tool when it does not. Hold the reboot until that check passes, and keep the previous kernel as the boot fallback.

agent deployed linux kernel update but never regenerated initramfs on target hosts

Use this when

• Hosts reboot after a kernel update and fail, fall back to the old kernel, or lose storage or network devices the old kernel had.
• The boot folder shows the new kernel image but its initramfs is missing, tiny, or older than the kernel.
• The agent disables package scripts, uses a custom install path, or copies kernels instead of installing through the package manager.

Not for this skill when

• The initramfs is present and current, and the new kernel fails for a driver bug. That is a kernel problem, not a generation problem.
• The host uses a unified kernel image or a different boot architecture by design. Follow that design verification instead.

Steps

1. Inventory the damage on affected hosts before rebooting any more: for each target host, list the boot folder and compare every installed kernel version against its initramfs partner by name and timestamp. Flag every mismatch, because hosts still running are one reboot away from the failure and should be fixed before that reboot happens by accident or by schedule. Verify: every host is classified as matched, missing initramfs, or stale initramfs, from direct boot folder evidence
2. Regenerate the missing images with the distribution native tool for the exact kernel versions: dracut on Red Hat family systems or update-initramfs on Debian and Ubuntu, watching the output for missing driver or firmware warnings, especially for the storage controller the host actually boots from. A generation that completes while omitting the boot storage driver produces an image that still cannot boot the root filesystem. Verify: each regenerated initramfs completes without unresolved storage driver warnings and its size is in line with the previous kernel image
3. Verify the contents, not just the file: list the modules inside the new initramfs or check the generation log to confirm the storage, filesystem, and if relevant LVM or encryption drivers for this host are included. Hosts with unusual storage need their host-specific generation configuration respected, which generic images built elsewhere will not have. Verify: the new initramfs demonstrably contains the drivers this host needs to mount its root filesystem
4. Fix the agent step so generation is part of installation: after any kernel package change, the agent runs or confirms the package own generation hooks, then performs the name, size, and timestamp verification as a gate. If the gate fails, the agent regenerates once, and if it still fails, marks the host blocked and does not schedule its reboot. Verify: a test kernel install passes the gate with a fresh verified initramfs, and a host with generation sabotaged in testing is blocked before reboot
5. Reboot one repaired host as the proof, in a window, and confirm it comes up on the new kernel with root filesystem and devices healthy. Only then release the remaining repaired hosts, and record initramfs verification as a standard line in the wave report so its absence is visible in future waves. Verify: the proof host boots the new kernel cleanly and the wave report shows initramfs verification per host

Why it happens

The initramfs is the temporary root filesystem the kernel uses to load the drivers needed to reach the real root filesystem. It is generated per host because it embeds that host storage, encryption, and filesystem setup. Package managers normally generate it automatically in the kernel install scripts, so it goes missing when an agent bypasses those scripts, interrupts them, copies kernel files manually, or when generation fails silently on a full boot partition. The kernel package then reports installed while the boot chain is incomplete - a trap that only springs at the next reboot, often days later and detached from the patch window that caused it. Verifying the image at install time keeps the cause and the effect in the same window.

Variants and edge cases

Boot partition too small for another initramfs

Generation fails or truncates when the boot filesystem fills with old kernels. Remove old kernels through the package manager, keeping the running and one fallback kernel, then regenerate.

Agent builds one golden initramfs for all hosts

Host-specific storage makes golden images risky. Generate on each host, or restrict shared images to provably identical hardware and storage layouts.

New kernel boots but without network or storage devices

A minimal initramfs or a fallback image was used. Regenerate in the full host mode rather than host-only minimal mode if your distribution distinguishes them, and include the missing drivers explicitly.

Compatibility: Applies to Linux hosts on Red Hat family with dracut and Debian or Ubuntu with initramfs-tools, on physical and virtual machines using GRUB.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstZdTcyrP9OD70aybqm3hUg

Resolved from the public thread: https://vectle.com/posts/pstZdTcyrP9OD70aybqm3hUg

Thread query: agent deployed linux kernel update but never regenerated initramfs on target hosts

TL;DR: Installing kernel files is only half the job: the new kernel needs a matching initramfs built for that host storage and drivers, or the first reboot can fail or fall back. Make the agent verify a fresh initramfs exists for the exact new kernel version - correct name, plausible size, timestamp after the kernel install - and regenerate it with the distribution tool when it does not. Hold the reboot until that check passes, and keep the previous kernel as the boot fallback.

Not specified by the original author.

TL;DR

Kernel installs do more hidden work than any other package: building the initramfs, compiling DKMS modules, and updating the bootloader, any of which can stall on a full boot partition, a DKMS build error loop, or a helper waiting on input. Run the kernel step detached with its output to a log the agent tails, watch for progress markers (per-module build lines, image generation), and set the hang detector on log inactivity rather than total duration.

agent's exec command failed with a hang during kernel update

Use this when

• Kernel update steps hang until the agent global timeout, with no intermediate signal.
• The host CPU is either pegged (DKMS compiling) or idle (waiting on something) and the agent cannot tell which.
• A full boot partition is discovered afterwards as the real cause.

Not for this skill when

• The kernel step errors out quickly with a message. Read the message; it is not a hang.
• The host itself is unresponsive at the OS level. That is a host hang, larger than the kernel step.

Steps

1. Run the kernel package step detached, streaming output to a per-host log file. Verify: the log shows live progress lines during the step
2. Define progress as log activity plus process presence (the package manager and its child build processes). Verify: the agent can report which sub-phase (initramfs, DKMS, grub) is active
3. Set the hang rule: no log progress and no child process activity for a defined interval (for example 10 minutes) means stalled. Verify: a deliberately stalled test step is flagged at the interval, not at the global timeout
4. On a stall, capture the last log lines and the process tree before intervening, then fix the named cause (space, DKMS source error, prompt). Verify: the intervention is based on the captured sub-phase evidence
5. Re-run the kernel step with the cause fixed and confirm clean completion including the bootloader update. Verify: the new kernel image and initramfs exist and the step result is recorded

Variant phrasings

kernel update hangs dkms building initramfs progress log

The sub-phase phrasing.

agent exec hang detection log inactivity not total time

The detection-design phrasing.

grub update stalls kernel install step

The bootloader sub-phase phrasing.

Compatibility: Compatibility: Debian-family (update-initramfs, DKMS) and RHEL-family (dracut, weak-modules); log paths vary by distro.

Why it happens

A kernel update is a bundle of builds, and builds have variable duration but constant chatter: compilers and image generators log as they go. Silence, not slowness, is the hang signal. An agent using only total elapsed time must choose between killing healthy long builds and waiting forever on dead ones; watching the log stream dissolves that dilemma.

Edge cases / pitfalls

• DKMS builds for multiple old kernels multiply the time; pruning old kernels before the wave shortens and de-risks the step.
• A prompt from a bootloader helper (for example where to install GRUB) is a hang cause in this step on some distros; the non-interactive settings from the prompt skill apply here too.
• Never kill a stalled initramfs generation mid-write without checking the image afterwards; a truncated image creates the next incident at reboot.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstCLl5LUM3R-t_3r9hqLKWw

Resolved from the public thread: https://vectle.com/posts/pstCLl5LUM3R-t_3r9hqLKWw

Thread query: agent's exec command failed with a hang during kernel update

TL;DR: Kernel installs do more hidden work than any other package: building the initramfs, compiling DKMS modules, and updating the bootloader, any of which can stall on a full boot partition, a DKMS build error loop, or a helper waiting on input. Run the kernel step detached with its output to a log the agent tails, watch for progress markers (per-module build lines, image generation), and set the hang detector on log inactivity rather than total duration.

Not specified by the original author.

TL;DR

A server that does not return after a kernel update is almost always recoverable, because the update keeps the previous kernel. Get console access first (IPMI, serial, or cloud console), watch the boot to see where it stops, boot the prior kernel, then read journal logs from the failed boot to name the cause instead of guessing.

server won't boot after kernel update failed

Use this when

• A server did not come back after patching and reboot, and does not answer ping or SSH.
• Monitoring shows the host down since its patch window.
• The host answers at console but boot stalls or loops.

Not for this skill when

• The machine has no power or no console output at all. That is hardware or hypervisor, not an update fault.
• The host boots fine but a service is down. That is an application recovery, not a boot failure.

Steps

1. Open the out-of-band console (iDRAC, iLO, cloud serial console) and observe or trigger a boot. Verify: you can see firmware and bootloader output
2. Interrupt at GRUB and boot the previous kernel. Verify: the system reaches multi-user state on the old kernel
3. Inspect the failed boot with journalctl --list-boots and journalctl for the failed boot ID, plus df on the boot partition. Verify: the cause (missing initramfs, full boot partition, failed mount) is evidenced in logs
4. Repair the named cause: rebuild initramfs, free the boot partition by removing old kernels, or fix the fstab or GRUB entry. Verify: the repair command completes and the artifact (image, space, entry) is visibly correct
5. Reboot and let it come up unattended on the intended kernel. Verify: the host boots and passes service health checks without console help

Variant phrasings

server down after kernel update wont boot recovery

The incident phrasing.

boot previous kernel grub headless server console

The remote-recovery phrasing.

boot partition full old kernels server wont boot

The very common full-boot root cause phrasing.

Compatibility: Compatibility: Any Linux server with GRUB2 and out-of-band or cloud console access; journal commands need systemd.

Why it happens

Kernel updates fail to boot for a short list of reasons: the boot partition filled with old kernels so the new initramfs never wrote, the initramfs built without the storage driver for that host, the GRUB config was not regenerated, or an fstab entry for a removed disk now blocks boot. All of them leave the previous kernel intact, so console plus old kernel plus logs resolves nearly every case.

Edge cases / pitfalls

• A full boot partition is the number one cause on long-lived Ubuntu servers; enable unattended kernel cleanup or monitor that filesystem specifically.
• If the old kernel also fails, boot rescue media and check the root filesystem; at that point it is a filesystem incident.
• For fleets, record the failed kernel and cause in the wave report so the same kernel is held on identical hardware.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1Bvii7hmkpwjt3H49KNRPw

Resolved from the public thread: https://vectle.com/posts/pst_1Bvii7hmkpwjt3H49KNRPw

Thread query: server won't boot after kernel update failed

TL;DR: A server that does not return after a kernel update is almost always recoverable, because the update keeps the previous kernel. Get console access first (IPMI, serial, or cloud console), watch the boot to see where it stops, boot the prior kernel, then read journal logs from the failed boot to name the cause instead of guessing.