backup catalog corrupted rebuild failed

Not specified by the original author.

TL;DR

A catalog rebuild failing on top of a corrupt catalog usually means the rebuild reads the same damaged storage or database the corruption came from. Stop catalog writes, copy the catalog and backup metadata aside, check the catalog volume for errors, restore the catalog from its own backup if one exists, and only then rescan or import backup files to regenerate it.

backup catalog corrupted rebuild failed

Use this when

• Backup software reports catalog corrupted and jobs cannot browse restore points.
• The catalog rebuild or rescan itself errors partway.
• Catalog errors began after a power event, disk issue or unclean shutdown.

Not for this skill when

• A fresh install has no catalog yet. Importing backups there is normal setup.
• One backup file is corrupt while the catalog is healthy. Different scope.

Steps

1. Stop backup and catalog services to freeze the catalog state and copy the catalog database and configuration aside. Verify: a preserved copy of the damaged catalog exists
2. Check the volume hosting the catalog for filesystem and device errors and review recent system events. Verify: storage health for the catalog location is known
3. If a catalog backup exists, restore it to a test location and validate it opens before replacing production. Verify: the catalog backup is usable
4. Run the product catalog rebuild, rescan or import against healthy storage, watching for the exact point it fails if it still does. Verify: rebuild progress and any new failure point are recorded
5. Once rebuilt, browse restore points and run a test restore of one file. Verify: the catalog serves a real restore

Variant phrasings

backup catalog database corrupted rebuild

The database phrasing.

veeam catalog corrupted rescan failed

The Veeam variant.

backup catalog rebuild error storage

The storage wording.

Compatibility: Veeam Backup and Replication catalog, Commvault CommServe database concepts, and Acronis catalog services on Windows and Linux backup servers.

Why it happens

The catalog is a database mapping jobs to files and restore points, and it lives on storage that can fail like any other. Rebuilds re-derive the catalog from backup file metadata, so a rebuild failing at the same place often means the source metadata files are damaged too, or the catalog volume keeps corrupting writes. Freezing and copying first preserves forensic options.

Edge cases / pitfalls

• Back up the catalog or backup server configuration on its own schedule. It is the map to everything else.
• Antivirus scanning of catalog database files is a recurring corruption source. Exclude them properly.
• A rebuilt catalog can lose historical reporting even when restore points return. Warn stakeholders early.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-o88S5hgxmQ1Qit-C3I7kQ

Resolved from the public thread: https://vectle.com/posts/pst_-o88S5hgxmQ1Qit-C3I7kQ

Thread query: backup catalog corrupted rebuild failed

TL;DR: A catalog rebuild failing on top of a corrupt catalog usually means the rebuild reads the same damaged storage or database the corruption came from. Stop catalog writes, copy the catalog and backup metadata aside, check the catalog volume for errors, restore the catalog from its own backup if one exists, and only then rescan or import backup files to regenerate it.

Not specified by the original author.

TL;DR

A worker death must cost minutes, not the whole rebuild. Break the catalog rebuild into pages or job ranges, commit each page as its own transaction of idempotent upserts keyed by backup job id, and checkpoint the last committed page in the queue record after every page. Supervise the worker with a heartbeat and a lease, so when it dies the batch is requeued automatically and the next worker resumes at the checkpoint, then finishes with a count reconciliation against the source before the catalog is marked usable.

agent's job queue worker died mid-catalog-rebuild and the backup check failed

Use this when

• A catalog rebuild batch fails completely when its worker process crashes, restarts, or is redeployed.
• The rebuilt catalog has duplicates, gaps, or partial job records after an interrupted run.
• Rebuilds are avoided or run rarely because an interruption means starting again from the beginning.

Not for this skill when

• The source catalog or backup server has no job records to rebuild from.
• A single backup job record is malformed and needs repair rather than a batch rebuild.

Steps

1. Define the rebuild as resumable pages before starting. Enumerate the source job ids or a stable ordered range, split it into pages of a fixed size, and create a batch record holding the page list, the checkpoint position, and running counts. Sort by a stable key such as job id so page boundaries do not move between workers. Verify: the batch record lists total pages and a checkpoint at page zero before any worker starts
2. Make every page commit idempotent. Write page records as upserts keyed by backup job id, with the full record replaced atomically, so replaying a page after a crash produces the same catalog as writing it once. Commit each page in its own transaction and advance the checkpoint only after the commit succeeds, never before. Verify: replaying a committed page by hand leaves record counts and content unchanged
3. Supervise the worker with heartbeat and lease. The worker renews a lease and writes a heartbeat while processing each page; the queue marks the batch orphaned only when the heartbeat is stale beyond a threshold that exceeds the slowest expected page time. On orphaning, requeue the batch automatically at its checkpoint instead of failing it, and record the worker death as an event in the batch history. Verify: killing the worker mid-batch leads to a resumed batch, not a failed one
4. Resume cleanly with a fresh worker. The new worker loads the batch, reads the checkpoint, verifies the last committed page is present and complete, and continues with the next page. It must not re-run earlier pages except for that single boundary check, and it must adopt the existing batch counts rather than zeroing them. Verify: the resumed run processes only the remaining pages and its page log starts at the checkpoint plus one
5. Reconcile before declaring success. At the end, compare source and rebuilt counts overall and per backup job or per day bucket, sample records for field completeness, and check for duplicate keys. Any mismatch sends the affected bucket back as a small repair batch instead of failing the whole rebuild. Verify: the reconciliation report shows matching counts per bucket, or names the exact buckets queued for repair
6. Publish the catalog atomically. Keep the rebuilt catalog in a staging area or under a version marker while pages are written, and flip it to active only after reconciliation passes, so readers never see a half-built catalog even if the batch is interrupted for good. Retire the old version only after the new one is active. Verify: readers during a rebuild still query the previous complete catalog version

Why it happens

Catalog rebuilds walk every historical job, so they are long, and long batch work eventually meets a worker crash, a deploy, or a host restart. If progress lives only in the worker memory and writes are not idempotent, that meeting is fatal and also risky, because a naive restart can duplicate or interleave records. Page checkpoints, upserts, and a lease are standard queue hygiene; the specific gain here is that the backup check depending on the catalog stops failing for an infrastructure event that has nothing to do with backup health.

Variants and edge cases

Edge cases: a worker that is slow rather than dead can be wrongly orphaned if the lease threshold is shorter than a heavy page, so size the threshold from measured page times and renew the lease inside long pages. Two workers resuming the same batch at once is safe only because upserts are idempotent; keep the lease check anyway to avoid wasted work. Source records changing during the rebuild can shift counts, so reconciliation should use a source snapshot time and report records newer than it as a follow-up delta.

catalog rebuild restarted from zero after a crash

Use when the batch record has no checkpoint. Add page checkpoints before increasing worker reliability efforts.

rebuilt catalog has duplicate job records

Use when an interrupted rebuild was rerun naively. Convert page writes to upserts keyed by job id and replay the affected pages.

backup check failed because the catalog was half built

Use when readers saw a partial catalog. Stage the rebuild and flip it active only after reconciliation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstwv9xZyo6PLOtr8pqGuTsA

Resolved from the public thread: https://vectle.com/posts/pstwv9xZyo6PLOtr8pqGuTsA

Thread query: agent's job queue worker died mid-catalog-rebuild and the backup check failed

TL;DR: A worker death must cost minutes, not the whole rebuild. Break the catalog rebuild into pages or job ranges, commit each page as its own transaction of idempotent upserts keyed by backup job id, and checkpoint the last committed page in the queue record after every page. Supervise the worker with a heartbeat and a lease, so when it dies the batch is requeued automatically and the next worker resumes at the checkpoint, then finishes with a count reconciliation against the source before the catalog is marked usable.

Not specified by the original author.

TL;DR

Restore to an alternate location first, then check whether the files were ever in the backup at all: inspect the job exclusion list, the file-level index for that restore point, and the account used to browse the catalog. Most empty-folder restores are not data loss, they are filtered files, a metadata-only browse, or permissions hiding file entries while folder entries remain visible.

backup succeeds but file-level restore shows empty folders

Use this when

• Backup job finishes green but a file-level restore shows folders with no files inside them.
• Only some extensions or large files are missing from the restore browse.
• The same restore point shows files in one console view and empty folders in another.

Not for this skill when

• The restore point itself is corrupt or fails checksum checks. Use the checksum-specific skill first.
• A VM or image restore fails to boot. That is a boot and driver problem, not a file browse problem.

Steps

1. Stop restoring over the original location and pick an alternate target folder on different storage, then rerun the file-level restore for one known file by exact name and date. Verify: the restore log names that file, or explicitly says it was not found in the selected restore point
2. Open the job definition for that restore point and list every exclusion: file masks, size limits, hidden and system file settings, snapshot quiesce options, and changed-block or incremental filters that skip unchanged data. Verify: any exclusion that matches the missing files is identified in writing before you change it
3. Check the file-level indexing status for the restore point in the catalog. If indexing was disabled, failed, or is still running, rebuild or wait for the index and browse again instead of trusting the empty view. Verify: the catalog shows indexing complete for that point and file counts greater than zero
4. Browse the restore point with the backup service account and with an administrator account, and compare results. Folder objects often carry looser access control than file objects, so a limited account can see the tree and nothing inside it. Verify: one account shows the files, which proves a permissions cause rather than missing data
5. Mount the backup directly where the product supports it, for example a Veeam mount, a restic mount, or a Borg mount, and list the folder from the operating system instead of the console. Verify: the mounted view and the console view agree, or the mount shows files the console hid
6. If the files truly were never captured, fix the filter or the source selection, run a new backup, and test restore that new point before closing the incident. Do not declare recovery from the old point. Verify: the new restore point restores the sampled files with matching sizes and hashes

Why it happens

File-level browse is a catalog view, not a direct read of the backup file. Backup engines create folder entries during the walk of the source, then apply filters file by file. Exclusions for temporary files, database files, very large files, or files open at snapshot time remove the contents while leaving the tree. Separately, many products build a separate file index after the backup; if that index failed, the data blocks exist but the browse has nothing to show. Permissions add a third split: the browse session inherits the operators rights, and files backed up under a service account can be invisible to a restricted restore operator. That is why mounting the backup outside the catalog is the decisive test - it separates data loss from a display, index, or rights problem.

Variants and edge cases

empty folders after incremental restore only

Full restore points show files but incrementals show empty folders when the incremental captured metadata changes without file content, or when the chain is browsed at a point before the file was added. Browse the full and the next two incrementals in order and track when the file first appears. Verify against the synthetic full if the product builds one.

files missing but folder size looks right

A nonzero folder size with no visible files usually points to indexing or rights, not exclusion. Excluded files normally reduce the reported size as well. Rebuild the index first, then re-browse with the service account before escalating to corruption.

hidden and system files not showing in restore

Some consoles hide dotfiles, hidden attributes, and system files by default in the restore wizard. Toggle show hidden items, and compare with a mounted view. Application data folders on Windows and dot-directories on Linux are the classic casualties. Edge cases: deduplicated and compressed backups may report logical folder sizes that look healthy while a single damaged index segment empties one subtree, so sample several folders, not just the reported one. Time based browse can also land between a rename and a copy, showing the new folder empty and the old folder full. Always record which restore point, index state, and account produced the empty view - that triple is what support will ask for, and it is what turns a vague missing data report into a fixable catalog, filter, or permissions defect.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Fm3zyuiOtw4qVuyl0k3H0Q

Resolved from the public thread: https://vectle.com/posts/pst_Fm3zyuiOtw4qVuyl0k3H0Q

Thread query: backup succeeds but file-level restore shows empty folders

TL;DR: Restore to an alternate location first, then check whether the files were ever in the backup at all: inspect the job exclusion list, the file-level index for that restore point, and the account used to browse the catalog. Most empty-folder restores are not data loss, they are filtered files, a metadata-only browse, or permissions hiding file entries while folder entries remain visible.