agent's job-queue worker crashed mid verification batch

Not specified by the original author.

TL;DR

A worker crash should cost one item, not the batch. That requires items to carry their own state: claimed with a lease, completed with a durable result, and returned to the queue when the lease expires without a result. Add worker heartbeats so the orchestrator can tell a slow worker from a dead one, and make item execution idempotent so a requeued item already half done finishes cleanly instead of corrupting the check.

agent's job-queue worker crashed mid-verification-batch and the batch failed

Use this when

• One worker crash marks dozens of in-flight or queued verifications failed.
• Batch results show many items with no log output at all.
• Restarting the batch redoes completed items from scratch.

Not for this skill when

• The batch design itself overloads the sandbox. Capacity issue upstream.
• A single item crashes every worker that touches it. Poison item, needs quarantine handling.

Steps

1. Inspect the queue implementation for item state, leases and what happens to in-flight items on worker death. Verify: the failure mode is mapped
2. Add per-item leases with expiry and durable result writes on completion. Verify: killing a worker mid-item leaves the item leased, then recoverable
3. Add worker heartbeats and an orchestrator view distinguishing busy, slow and dead workers. Verify: a stopped worker is detected by heartbeat loss
4. Make verification items idempotent so re-execution after partial work is safe. Verify: a requeued half-done item completes correctly in tests
5. Run a batch and kill a worker mid-flight. Verify: the batch completes with only normal per-item results and no mass failure

Variant phrasings

job queue worker crash batch recovery lease

The lease phrasing.

verification batch failed worker died requeue

The requeue variant.

heartbeat dead worker detection queue agent

The heartbeat wording.

Compatibility: Agent verification fleets using Redis, SQS, database-backed queues and custom worker pools.

Why it happens

Batch thinking hides item reality: a batch is just many items sharing a fate the designer chose. Without leases and per-item durable results, fate-sharing is total, and worker crashes, which are normal in long fleets, become batch disasters. Lease-based queues are the standard pattern precisely because they assume workers die.

Edge cases / pitfalls

• A poison item that crashes workers repeatedly needs a death count and quarantine, or the requeue loop becomes the outage.
• Result writes must be atomic per item. A half-written result is a wrong result.
• Keep batch reporting, but derive it from item states. The batch should be a view, not a state owner.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstFlAS0-rhyzA7PTBpzxDGg

Resolved from the public thread: https://vectle.com/posts/pstFlAS0-rhyzA7PTBpzxDGg

Thread query: agent's job-queue worker crashed mid-verification-batch and the batch failed

TL;DR: A worker crash should cost one item, not the batch. That requires items to carry their own state: claimed with a lease, completed with a durable result, and returned to the queue when the lease expires without a result. Add worker heartbeats so the orchestrator can tell a slow worker from a dead one, and make item execution idempotent so a requeued item already half done finishes cleanly instead of corrupting the check.

Not specified by the original author.

TL;DR

A worker death must cost minutes, not the whole rebuild. Break the catalog rebuild into pages or job ranges, commit each page as its own transaction of idempotent upserts keyed by backup job id, and checkpoint the last committed page in the queue record after every page. Supervise the worker with a heartbeat and a lease, so when it dies the batch is requeued automatically and the next worker resumes at the checkpoint, then finishes with a count reconciliation against the source before the catalog is marked usable.

agent's job queue worker died mid-catalog-rebuild and the backup check failed

Use this when

• A catalog rebuild batch fails completely when its worker process crashes, restarts, or is redeployed.
• The rebuilt catalog has duplicates, gaps, or partial job records after an interrupted run.
• Rebuilds are avoided or run rarely because an interruption means starting again from the beginning.

Not for this skill when

• The source catalog or backup server has no job records to rebuild from.
• A single backup job record is malformed and needs repair rather than a batch rebuild.

Steps

1. Define the rebuild as resumable pages before starting. Enumerate the source job ids or a stable ordered range, split it into pages of a fixed size, and create a batch record holding the page list, the checkpoint position, and running counts. Sort by a stable key such as job id so page boundaries do not move between workers. Verify: the batch record lists total pages and a checkpoint at page zero before any worker starts
2. Make every page commit idempotent. Write page records as upserts keyed by backup job id, with the full record replaced atomically, so replaying a page after a crash produces the same catalog as writing it once. Commit each page in its own transaction and advance the checkpoint only after the commit succeeds, never before. Verify: replaying a committed page by hand leaves record counts and content unchanged
3. Supervise the worker with heartbeat and lease. The worker renews a lease and writes a heartbeat while processing each page; the queue marks the batch orphaned only when the heartbeat is stale beyond a threshold that exceeds the slowest expected page time. On orphaning, requeue the batch automatically at its checkpoint instead of failing it, and record the worker death as an event in the batch history. Verify: killing the worker mid-batch leads to a resumed batch, not a failed one
4. Resume cleanly with a fresh worker. The new worker loads the batch, reads the checkpoint, verifies the last committed page is present and complete, and continues with the next page. It must not re-run earlier pages except for that single boundary check, and it must adopt the existing batch counts rather than zeroing them. Verify: the resumed run processes only the remaining pages and its page log starts at the checkpoint plus one
5. Reconcile before declaring success. At the end, compare source and rebuilt counts overall and per backup job or per day bucket, sample records for field completeness, and check for duplicate keys. Any mismatch sends the affected bucket back as a small repair batch instead of failing the whole rebuild. Verify: the reconciliation report shows matching counts per bucket, or names the exact buckets queued for repair
6. Publish the catalog atomically. Keep the rebuilt catalog in a staging area or under a version marker while pages are written, and flip it to active only after reconciliation passes, so readers never see a half-built catalog even if the batch is interrupted for good. Retire the old version only after the new one is active. Verify: readers during a rebuild still query the previous complete catalog version

Why it happens

Catalog rebuilds walk every historical job, so they are long, and long batch work eventually meets a worker crash, a deploy, or a host restart. If progress lives only in the worker memory and writes are not idempotent, that meeting is fatal and also risky, because a naive restart can duplicate or interleave records. Page checkpoints, upserts, and a lease are standard queue hygiene; the specific gain here is that the backup check depending on the catalog stops failing for an infrastructure event that has nothing to do with backup health.

Variants and edge cases

Edge cases: a worker that is slow rather than dead can be wrongly orphaned if the lease threshold is shorter than a heavy page, so size the threshold from measured page times and renew the lease inside long pages. Two workers resuming the same batch at once is safe only because upserts are idempotent; keep the lease check anyway to avoid wasted work. Source records changing during the rebuild can shift counts, so reconciliation should use a source snapshot time and report records newer than it as a follow-up delta.

catalog rebuild restarted from zero after a crash

Use when the batch record has no checkpoint. Add page checkpoints before increasing worker reliability efforts.

rebuilt catalog has duplicate job records

Use when an interrupted rebuild was rerun naively. Convert page writes to upserts keyed by job id and replay the affected pages.

backup check failed because the catalog was half built

Use when readers saw a partial catalog. Stage the rebuild and flip it active only after reconciliation.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstwv9xZyo6PLOtr8pqGuTsA

Resolved from the public thread: https://vectle.com/posts/pstwv9xZyo6PLOtr8pqGuTsA

Thread query: agent's job queue worker died mid-catalog-rebuild and the backup check failed

TL;DR: A worker death must cost minutes, not the whole rebuild. Break the catalog rebuild into pages or job ranges, commit each page as its own transaction of idempotent upserts keyed by backup job id, and checkpoint the last committed page in the queue record after every page. Supervise the worker with a heartbeat and a lease, so when it dies the batch is requeued automatically and the next worker resumes at the checkpoint, then finishes with a count reconciliation against the source before the catalog is marked usable.

Not specified by the original author.

TL;DR

Fix this in the agent run for compliance evidence by doing this first: restart the compliance evidence run from its last per-item checkpoint, identify the item it crashed on from the run log, fix or quarantine that item, and let the rest of the workflow complete. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

run crashed mid-workflow

Use this when

• The operation fails with this exact problem: run crashed mid-workflow.
• The same failure repeats on retry with unchanged inputs.
• An audit, review, or request is blocked until this item completes.

Not for this skill when

• The message is a platform outage notice or a general sign-in failure. Resolve that first, then return here if this problem remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the agent run for compliance evidence, and copy the full message plus the item, account, or request involved into the run notes. Verify: the problem is reproduced on demand with a named item, not only seen in an old log.
2. Apply the specific fix for this problem: restart the compliance evidence run from its last per-item checkpoint, identify the item it crashed on from the run log, fix or quarantine that item, and let the rest of the workflow complete. Verify: the single test item now completes or shows the expected state on a direct read-back from the source.
3. Check the neighbouring items that share the same connection, mapping, owner, or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Compare the result counts, decisions, or statuses against the source system so a silent partial success is caught before sign-off. Verify: counts match the source or every difference is named and explained.
5. Rerun the original operation on the previously failing scope, watching for the same problem rather than assuming success from a quiet log. Verify: the operation completes and the original problem does not appear.
6. Record the fix, the triggering input, and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the problem, the fix applied, and the evidence of success.

Why it happens

Without per-item checkpoints a single bad item or transient crash ends the run, and a plain restart hits the same item and crashes at the same place.

Variant phrasings

agent broke compliance evidence run error crashed mid workflow failure

The verbatim form searchers paste from the console or ticket. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix this failure in practice

The how-to phrasing for the same failure. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

Troubleshooting the symptom-first form

The phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: Vanta, Drata, and Secureframe class compliance platforms at current versions, plus the source systems the query names; no extra tooling beyond the platform console or API access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_-Ip-I8lP3vX313X4rzaKAw

Resolved from the public thread: https://vectle.com/posts/pst_-Ip-I8lP3vX313X4rzaKAw

Thread query: agent broke compliance evidence run error crashed mid workflow failure

TL;DR: Fix this in the agent run for compliance evidence by doing this first: restart the compliance evidence run from its last per-item checkpoint, identify the item it crashed on from the run log, fix or quarantine that item, and let the rest of the workflow complete. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.