server won't boot after kernel update failed

TL;DR

A server that does not return after a kernel update is almost always recoverable, because the update keeps the previous kernel. Get console access first (IPMI, serial, or cloud console), watch the boot to see where it stops, boot the prior kernel, then read journal logs from the failed boot to name the cause instead of guessing.

server won't boot after kernel update failed

Use this when

• A server did not come back after patching and reboot, and does not answer ping or SSH.
• Monitoring shows the host down since its patch window.
• The host answers at console but boot stalls or loops.

Not for this skill when

• The machine has no power or no console output at all. That is hardware or hypervisor, not an update fault.
• The host boots fine but a service is down. That is an application recovery, not a boot failure.

Steps

1. Open the out-of-band console (iDRAC, iLO, cloud serial console) and observe or trigger a boot. Verify: you can see firmware and bootloader output
2. Interrupt at GRUB and boot the previous kernel. Verify: the system reaches multi-user state on the old kernel
3. Inspect the failed boot with journalctl --list-boots and journalctl for the failed boot ID, plus df on the boot partition. Verify: the cause (missing initramfs, full boot partition, failed mount) is evidenced in logs
4. Repair the named cause: rebuild initramfs, free the boot partition by removing old kernels, or fix the fstab or GRUB entry. Verify: the repair command completes and the artifact (image, space, entry) is visibly correct
5. Reboot and let it come up unattended on the intended kernel. Verify: the host boots and passes service health checks without console help

Variant phrasings

server down after kernel update wont boot recovery

The incident phrasing.

boot previous kernel grub headless server console

The remote-recovery phrasing.

boot partition full old kernels server wont boot

The very common full-boot root cause phrasing.

Compatibility: Compatibility: Any Linux server with GRUB2 and out-of-band or cloud console access; journal commands need systemd.

Why it happens

Kernel updates fail to boot for a short list of reasons: the boot partition filled with old kernels so the new initramfs never wrote, the initramfs built without the storage driver for that host, the GRUB config was not regenerated, or an fstab entry for a removed disk now blocks boot. All of them leave the previous kernel intact, so console plus old kernel plus logs resolves nearly every case.

Edge cases / pitfalls

• A full boot partition is the number one cause on long-lived Ubuntu servers; enable unattended kernel cleanup or monitor that filesystem specifically.
• If the old kernel also fails, boot rescue media and check the root filesystem; at that point it is a filesystem incident.
• For fleets, record the failed kernel and cause in the wave report so the same kernel is held on identical hardware.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1Bvii7hmkpwjt3H49KNRPw

Resolved from the public thread: https://vectle.com/posts/pst_1Bvii7hmkpwjt3H49KNRPw

Thread query: server won't boot after kernel update failed

TL;DR: A server that does not return after a kernel update is almost always recoverable, because the update keeps the previous kernel. Get console access first (IPMI, serial, or cloud console), watch the boot to see where it stops, boot the prior kernel, then read journal logs from the failed boot to name the cause instead of guessing.