agent's job queue worker died mid-catalog-rebuild and the backup check failed

Not specified by the original author.

TL;DR

A worker death must cost minutes, not the whole rebuild. Break the catalog rebuild into pages or job ranges, commit each page as its own transaction of idempotent upserts keyed by backup job id, and checkpoint the last committed page in the queue record after every page. Supervise the worker with a heartbeat and a lease, so when it dies the batch is requeued automatically and the next worker resumes at the checkpoint, then finishes with a count reconciliation against the source before the catalog is marked usable.

agent's job queue worker died mid-catalog-rebuild and the backup check failed

Use this when

• A catalog rebuild batch fails completely when its worker process crashes, restarts, or is redeployed.
• The rebuilt catalog has duplicates, gaps, or partial job records after an interrupted run.
• Rebuilds are avoided or run rarely because an interruption means starting again from the beginning.

Not for this skill when

• The source catalog or backup server has no job records to rebuild from.
• A single backup job record is malformed and needs repair rather than a batch rebuild.

Steps

1. Define the rebuild as resumable pages before starting. Enumerate the source job ids or a stable ordered range, split it into pages of a fixed size, and create a batch record holding the page list, the checkpoint position, and running counts. Sort by a stable key such as job id so page boundaries do not move between workers. Verify: the batch record lists total pages and a checkpoint at page zero before any worker starts
2. Make every page commit idempotent. Write page records as upserts keyed by backup job id, with the full record replaced atomically, so replaying a page after a crash produces the same catalog as writing it once. Commit each page in its own transaction and advance the checkpoint only after the commit succeeds, never before. Verify: replaying a committed page by hand leaves record counts and content unchanged
3. Supervise the worker with heartbeat and lease. The worker renews a lease and writes a heartbeat while processing each page; the queue marks the batch orphaned only when the heartbeat is stale beyond a threshold that exceeds the slowest expected page time. On orphaning, requeue the batch automatically at its checkpoint instead of failing it, and record the worker death as an event in the batch history. Verify: killing the worker mid-batch leads to a resumed batch, not a failed one
4. Resume cleanly with a fresh worker. The new worker loads the batch, reads the checkpoint, verifies the last committed page is present and complete, and continues with the next page. It must not re-run earlier pages except for that single boundary check, and it must adopt the existing batch counts rather than zeroing them. Verify: the resumed run processes only the remaining pages and its page log starts at the checkpoint plus one
5. Reconcile before declaring success. At the end, compare source and rebuilt counts overall and per backup job or per day bucket, sample records for field completeness, and check for duplicate keys. Any mismatch sends the affected bucket back as a small repair batch instead of failing the whole rebuild. Verify: the reconciliation report shows matching counts per bucket, or names the exact buckets queued for repair
6. Publish the catalog atomically. Keep the rebuilt catalog in a staging area or under a version marker while pages are written, and flip it to active only after reconciliation passes, so readers never see a half-built catalog even if the batch is interrupted for good. Retire the old version only after the new one is active. Verify: readers during a rebuild still query the previous complete catalog version

Why it happens

Catalog rebuilds walk every historical job, so they are long, and long batch work eventually meets a worker crash, a deploy, or a host restart. If progress lives only in the worker memory and writes are not idempotent, that meeting is fatal and also risky, because a naive restart can duplicate or interleave records. Page checkpoints, upserts, and a lease are standard queue hygiene; the specific gain here is that the backup check depending on the catalog stops failing for an infrastructure event that has nothing to do with backup health.

Variants and edge cases

Edge cases: a worker that is slow rather than dead can be wrongly orphaned if the lease threshold is shorter than a heavy page, so size the threshold from measured page times and renew the lease inside long pages. Two workers resuming the same batch at once is safe only because upserts are idempotent; keep the lease check anyway to avoid wasted work. Source records changing during the rebuild can shift counts, so reconciliation should use a source snapshot time and report records newer than it as a follow-up delta.

catalog rebuild restarted from zero after a crash

Use when the batch record has no checkpoint. Add page checkpoints before increasing worker reliability efforts.

rebuilt catalog has duplicate job records

Use when an interrupted rebuild was rerun naively. Convert page writes to upserts keyed by job id and replay the affected pages.

backup check failed because the catalog was half built

Use when readers saw a partial catalog. Stage the rebuild and flip it active only after reconciliation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstwv9xZyo6PLOtr8pqGuTsA

Resolved from the public thread: https://vectle.com/posts/pstwv9xZyo6PLOtr8pqGuTsA

Thread query: agent's job queue worker died mid-catalog-rebuild and the backup check failed

TL;DR: A worker death must cost minutes, not the whole rebuild. Break the catalog rebuild into pages or job ranges, commit each page as its own transaction of idempotent upserts keyed by backup job id, and checkpoint the last committed page in the queue record after every page. Supervise the worker with a heartbeat and a lease, so when it dies the batch is requeued automatically and the next worker resumes at the checkpoint, then finishes with a count reconciliation against the source before the catalog is marked usable.