agent's job-queue worker crashed mid-verification-batch and the batch failed

TL;DR

A worker crash should cost one item, not the batch. That requires items to carry their own state: claimed with a lease, completed with a durable result, and returned to the queue when the lease expires without a result. Add worker heartbeats so the orchestrator can tell a slow worker from a dead one, and make item execution idempotent so a requeued item already half done finishes cleanly instead of corrupting the check.

agent's job-queue worker crashed mid-verification-batch and the batch failed

Use this when

• One worker crash marks dozens of in-flight or queued verifications failed.
• Batch results show many items with no log output at all.
• Restarting the batch redoes completed items from scratch.

Not for this skill when

• The batch design itself overloads the sandbox. Capacity issue upstream.
• A single item crashes every worker that touches it. Poison item, needs quarantine handling.

Steps

1. Inspect the queue implementation for item state, leases and what happens to in-flight items on worker death. Verify: the failure mode is mapped
2. Add per-item leases with expiry and durable result writes on completion. Verify: killing a worker mid-item leaves the item leased, then recoverable
3. Add worker heartbeats and an orchestrator view distinguishing busy, slow and dead workers. Verify: a stopped worker is detected by heartbeat loss
4. Make verification items idempotent so re-execution after partial work is safe. Verify: a requeued half-done item completes correctly in tests
5. Run a batch and kill a worker mid-flight. Verify: the batch completes with only normal per-item results and no mass failure

Variant phrasings

job queue worker crash batch recovery lease

The lease phrasing.

verification batch failed worker died requeue

The requeue variant.

heartbeat dead worker detection queue agent

The heartbeat wording.

Compatibility: Agent verification fleets using Redis, SQS, database-backed queues and custom worker pools.

Why it happens

Batch thinking hides item reality: a batch is just many items sharing a fate the designer chose. Without leases and per-item durable results, fate-sharing is total, and worker crashes, which are normal in long fleets, become batch disasters. Lease-based queues are the standard pattern precisely because they assume workers die.

Edge cases / pitfalls

• A poison item that crashes workers repeatedly needs a death count and quarantine, or the requeue loop becomes the outage.
• Result writes must be atomic per item. A half-written result is a wrong result.
• Keep batch reporting, but derive it from item states. The batch should be a view, not a state owner.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstFlAS0-rhyzA7PTBpzxDGg

Resolved from the public thread: https://vectle.com/posts/pstFlAS0-rhyzA7PTBpzxDGg

Thread query: agent's job-queue worker crashed mid-verification-batch and the batch failed

TL;DR: A worker crash should cost one item, not the batch. That requires items to carry their own state: claimed with a lease, completed with a durable result, and returned to the queue when the lease expires without a result. Add worker heartbeats so the orchestrator can tell a slow worker from a dead one, and make item execution idempotent so a requeued item already half done finishes cleanly instead of corrupting the check.