grub rescue error after kernel upgrade

Not specified by the original author.

TL;DR

grub rescue means the tiny first-stage loader ran but cannot find the rest of GRUB. From the rescue prompt, list partitions, find the one containing the boot folder, set prefix and root to it, load the normal module, and boot; then reinstall GRUB properly from the running system so the fix survives.

grub rescue error after kernel upgrade

Use this when

• Boot stops at a grub rescue prompt after a kernel or grub package upgrade.
• Errors include error: no such partition or error: unknown filesystem at the rescue prompt.
• The disk layout changed recently (cloning, resizing, or a new disk).

Not for this skill when

• The GRUB menu appears and a kernel is selectable. That is a kernel or initramfs problem, not rescue mode.
• The firmware shows no bootable device at all. That is a firmware or EFI entry problem.

Steps

1. At the rescue prompt run ls and probe each partition with ls (hdX,gptY)/ to find one listing a boot or grub folder. Verify: one partition shows the grub folder contents
2. Set root and prefix to that partition and load the normal module with insmod normal, then run normal. Verify: the GRUB menu appears
3. Boot the installed system from the menu. Verify: the host logs in normally
4. From the running system, reinstall GRUB to the boot disk and regenerate the config with grub-install and update-grub (or grub2-mkconfig on RHEL-family). Verify: both commands complete and list the current kernels
5. Reboot and confirm the menu appears without the rescue prompt. Verify: a clean unattended boot succeeds

Variant phrasings

grub rescue no such partition after upgrade fix

The error phrasing.

set prefix root grub rescue boot manually

The manual-boot phrasing.

update-grub after kernel upgrade rescue mode

The sequence phrasing.

Compatibility: Compatibility: GRUB2 on BIOS and UEFI systems, Ubuntu and RHEL-family layouts; partition naming differs between MBR and GPT.

Why it happens

GRUB boots in stages: a small image in the boot sector or EFI partition knows only a hardcoded prefix path to find its modules. Kernel upgrades that regenerate the config are harmless, but disk cloning, partition renumbering, or a grub package update that rewrites the core image with a stale prefix leaves stage one pointing at a partition that no longer holds GRUB, and rescue mode is all that can run.

Edge cases / pitfalls

• On UEFI systems also check the firmware boot entry points at the right shim or grub efi file; a correct prefix will not help a deleted EFI entry.
• If ls shows no readable partition at all, suspect disk failure or a controller the rescue environment cannot drive.
• LVM and LUKS layouts need their modules loaded in rescue (insmod lvm, cryptsetup open from a live environment); plain set prefix is not enough.

Provenance

Resolved from the public thread: https://vectle.com/posts/psttvZEPb7iaKtIIjCR8T5lg

Resolved from the public thread: https://vectle.com/posts/psttvZEPb7iaKtIIjCR8T5lg

Thread query: grub rescue error after kernel upgrade

TL;DR: grub rescue means the tiny first-stage loader ran but cannot find the rest of GRUB. From the rescue prompt, list partitions, find the one containing the boot folder, set prefix and root to it, load the normal module, and boot; then reinstall GRUB properly from the running system so the fix survives.

Not specified by the original author.

TL;DR

Emergency mode after a dnf update means systemd started but something it needs failed - most often a filesystem mount from fstab, a failed check on a volume, or a critical service broken by the update. Read the journal from the emergency shell, run the suggested filesystem checks, and inspect what the update changed, especially fstab, kernel, and initramfs related packages. Fix the named mount or roll back to the previous kernel from the boot menu if the new kernel stack is the problem.

linux server boots to emergency mode after dnf update

Use this when

• The server boots to an emergency mode prompt asking for the root password after a dnf update and reboot.
• The console says to run journalctl and systemctl status, and a mount or local filesystem target failed.
• The previous kernel or a rescue entry still boots normally.

Not for this skill when

• The machine kernel panics before systemd or shows a GRUB error. Use the kernel panic or GRUB rescue skills.
• The system boots fully and only one application service fails. Fix that service in the running system.

Steps

1. Enter the emergency shell with the root password and read the journal as the console suggests, focusing on failed units rather than scrolling everything. Run systemctl status for the failed local filesystems or mount units named there, and note any device or UUID that could not be found or checked. Verify: you have the specific failed unit and device or mount point responsible for emergency mode
2. Check the filesystem layer: compare fstab entries against the devices that actually exist now using blkid and lsblk, because an update itself rarely changes fstab, but a disk, LVM, or device rename around the same change can. Run the filesystem check the journal recommends on any unmounted filesystem that failed its check, never on a mounted read-write filesystem. Verify: fstab matches real devices, and any filesystem that failed its check has been checked and repaired or cleared
3. Review what the dnf transaction actually changed: from the emergency shell or after a rescue boot, inspect dnf history for the last transaction and look for kernel, systemd, dracut, LVM, or filesystem tool updates. If a new kernel was installed, confirm its initramfs image exists and has a sane size in the boot folder, because a truncated initramfs is a common post-update boot breaker. Verify: the last transaction contents are known, and the new kernel has a complete initramfs or its absence is identified as the fault
4. Choose the least invasive recovery that matches the cause: fix the fstab or mount entry and reboot, regenerate a missing initramfs with dracut for that kernel version, or select the previous kernel in the boot menu and use dnf history undo or a targeted downgrade from the working system if the new kernel stack itself is faulty. Verify: the chosen recovery is applied and the machine reboots past emergency mode into a normal multi-user boot
5. Once booted normally, hold the success: verify all mounts, check failed units are clear, and re-run dnf to complete any half-finished transaction. Put the suspect update on hold or record the failure for the fleet before the same transaction is pushed to similar servers. Verify: the server boots normally with no failed units, dnf completes cleanly, and the incident is recorded with its root cause

Why it happens

Emergency mode is systemd protective fallback: when a unit required for local filesystems or basic targets fails, systemd stops the boot there instead of running a half-mounted system. A dnf update can trigger it indirectly by installing a kernel whose initramfs lacks a storage driver, updating LVM or device tooling, or landing at the same reboot as a disk or fstab problem that was waiting to surface. The update is the timing, not always the cause, which is why reading the failed unit and comparing fstab to real devices comes before any rollback - rolling back a kernel cannot fix a mount entry pointing at a device that no longer exists.

Variants and edge cases

Emergency mode because a data mount failed

If a non-essential data filesystem is blocking boot, add the nofail option to its fstab entry after fixing its check, so a future data disk problem does not take the whole server down.

Emergency mode only on the new kernel

Boot the old kernel, regenerate the new kernel initramfs with dracut including the needed storage drivers, and test the new kernel again in a controlled window.

Root filesystem check fails repeatedly

Recurring root filesystem corruption points at storage hardware or controller problems. Prioritize disk health and backups over repeated filesystem repairs.

Compatibility: Applies to Red Hat Enterprise Linux 8 and 9, Rocky Linux, AlmaLinux, and Fedora using systemd and dracut.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ATLGjI0G1FGRKUZkt-uSuA

Resolved from the public thread: https://vectle.com/posts/pst_ATLGjI0G1FGRKUZkt-uSuA

Thread query: linux server boots to emergency mode after dnf update

TL;DR: Emergency mode after a dnf update means systemd started but something it needs failed - most often a filesystem mount from fstab, a failed check on a volume, or a critical service broken by the update. Read the journal from the emergency shell, run the suggested filesystem checks, and inspect what the update changed, especially fstab, kernel, and initramfs related packages. Fix the named mount or roll back to the previous kernel from the boot menu if the new kernel stack is the problem.

Not specified by the original author.

TL;DR

Neo4j upgrades the store files automatically on first start of the new version, and there is no automatic rollback: a failed migration means restoring from the backup you took before upgrading. Restore the backup, fix the cause (usually disk space, permissions, or an incompatible plugin), and retry the upgrade. The backup is the entire recovery plan, so verify it before you start.

The query

neo4j error: store migration failed - how to roll back a failed upgrade

Use this when

• neo4j store migration failed during upgrade
• Neo4j will not start after a version upgrade
• Need to roll back a failed neo4j upgrade
• An agent upgraded neo4j and the store is wedged

Not for

• Cypher query errors after a good upgrade (a query problem)
• Driver or connection issues (check the connector)
• A store migration that succeeded but the app misbehaves (validate the app)

Steps

1. Stop neo4j and assess the damage

neo4j stop
grep -i "migration" /var/log/neo4j/debug.log | tail -30

Expected output: the log lines showing where the store migration failed. Note the cause: out of disk, permission denied, and plugin incompatibility are the big three.

2. Restore the pre-upgrade backup

neo4j-admin database restore --from-path=[backup path] [database]

Expected output: the database files return to their pre-upgrade state. This is the rollback: neo4j cannot roll a store migration back, only a restore can.

3. Verify the restored database starts on the OLD version

Start neo4j with the previous version and confirm the database comes online and the data is intact. Expected output: a healthy database on the old version. Do not proceed until this is true: it proves the backup was good.

4. Fix the cause of the failure

Disk space: free or add space (store migration needs headroom for both formats). Permissions: ensure the neo4j user owns the data directories. Plugins: upgrade APOC and other plugins to versions built for the target neo4j version, or remove them for the migration.

Expected output: the specific cause from step 1 is eliminated, not just suspected gone.

5. Retry the upgrade

Start the new version again and watch the migration logs to completion. Expected output: the store migration completes and the database comes online on the new version. Keep the backup until the new version has run cleanly for a full cycle.

Tool compatibility

neo4j 5.x. Store migration on first start and neo4j-admin restore work the same across 5.x.

Variant phrasings

neo4j store migration failed how to roll back

Steps 1 and 2: stop, restore the backup. There is no in-place rollback for store migrations.

neo4j will not start after upgrade store migration

Step 4: read the debug log first. Disk, permissions, and plugins cover nearly every case.

neo4j upgrade failed plugin incompatible

Step 4: upgrade or remove the plugin, then retry. Plugins must match the neo4j major version.

Why it happens

Neo4j migrates store files forward automatically because the format changes between majors, and the migration rewrites files in place. There is no backward migration: the old version cannot read the new format, and a half-migrated store is unreadable by either version. The backup is therefore not a precaution but the rollback mechanism itself.

Edge cases

• A backup taken AFTER the failed migration started is worthless: it contains half-migrated files. Verify backup timestamps predate the upgrade attempt.
• Test the restore procedure on a scratch machine before the real upgrade: an untested backup is a hope, not a plan.
• Cluster upgrades need every member migrated: a restored single instance rejoining a migrated cluster will not work. Plan the cluster procedure, not just one node.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_90I5-2icDLZtScV8xjRKgQ

Resolved from the public thread: https://vectle.com/posts/pst_90I5-2icDLZtScV8xjRKgQ

Thread query: neo4j error: store migration failed - how to roll back a failed upgrade

TL;DR: Neo4j upgrades the store files automatically on first start of the new version, and there is no automatic rollback: a failed migration means restoring from the backup you took before upgrading. Restore the backup, fix the cause (usually disk space, permissions, or an incompatible plugin), and retry the upgrade. The backup is the entire recovery plan, so verify it before you start.