zpool scrub failed with unrecoverable errors

TL;DR

Unrecoverable errors after a scrub mean ZFS found data whose checksums fail on every copy in the pool, so redundancy could not heal it. List the affected files from zpool status, restore those files from backup or a replica, clear errors only after data is replaced, and investigate the devices that let all copies go bad.

zpool scrub failed with unrecoverable errors

Use this when

• zpool status shows permanent errors after a scrub.
• Specific files are listed with unrecoverable or permanent error markers.
• Applications error reading the same files the scrub names.

Not for this skill when

• A scrub is running slowly with no errors. Performance, not damage.
• A single device is faulted but data errors are zero. That is redundancy loss, handled differently.

Steps

1. Run zpool status with verbose error listing and record every affected file and device. Verify: the damaged file list is complete
2. Check device health, SMART data and recent pool events to find whether one device or several contributed. Verify: suspect devices are identified
3. Restore each damaged file from backup, replica or application source, replacing the bad copy in place. Verify: restored files read cleanly and applications open them
4. Replace or remove failing devices and let resilver complete if device faults caused the loss. Verify: resilver finishes and pool redundancy is restored
5. Run a fresh scrub and confirm zero errors before clearing the old error state in your records. Verify: the new scrub reports no permanent errors

Variant phrasings

zpool permanent errors files damaged scrub

The permanent-errors phrasing.

zfs unrecoverable errors restore files

The restore variant.

zpool status errors checksum unrecoverable

The status wording.

Compatibility: OpenZFS 2.1 and 2.2 pools in mirror, RAIDZ and single-device layouts on Linux and FreeBSD.

Why it happens

Scrub reads every block and checks it against its checksum, healing from a good copy when one exists. Permanent errors mean every copy of those blocks failed, which happens when multiple devices degrade silently, a single-device pool takes damage, or the same bad data was written to all copies by faulty memory. The file list in pool status is the recovery work queue.

Edge cases / pitfalls

• On single-device pools any corruption is permanent. Their backups matter most.
• Clearing errors with zpool clear before restoring files destroys your only damage list. Save it first.
• If damaged files are VM images or databases, restore whole images from backup rather than patching files.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstm33KRUDIpTMSsR8GJf8WQ

Resolved from the public thread: https://vectle.com/posts/pstm33KRUDIpTMSsR8GJf8WQ

Thread query: zpool scrub failed with unrecoverable errors

TL;DR: Unrecoverable errors after a scrub mean ZFS found data whose checksums fail on every copy in the pool, so redundancy could not heal it. List the affected files from zpool status, restore those files from backup or a replica, clear errors only after data is replaced, and investigate the devices that let all copies go bad.