agent's heartbeat missed during long checksum run and orchestrator killed the verification

TL;DR

The verification is healthy, its heartbeat is not, so separate the two. Run the heartbeat on its own timer that keeps firing while hashing blocks, hash large files in chunks and emit progress with bytes processed after each chunk, and checkpoint per-file state so a restart resumes instead of rehashing everything. Then set the orchestrator lease from expected duration for the data size, with a grace rule based on progress, so a run that is demonstrably advancing is not killed for being slow.

agent's heartbeat missed during long checksum run and orchestrator killed the verification

Use this when

• Orchestrated verification runs are killed or marked failed on the largest files or archives, while smaller ones pass.
• The orchestrator reports a missed heartbeat or a dead worker, but the worker logs show hashing was progressing.
• Re-running the same verification manually completes and the checksum matches.

Not for this skill when

• The checksum genuinely mismatches and the file fails when hashed anywhere.
• The worker process crashes for a memory or disk reason during hashing.

Steps

1. Move the heartbeat off the hashing path. Run heartbeats from a separate thread, task, or sidecar timer that fires on a fixed interval regardless of what the worker is doing, and include a simple alive marker plus the current file and bytes processed. Test this by hashing a deliberately huge file and confirming heartbeats continue at their normal interval throughout. Verify: heartbeat timestamps stay evenly spaced during a long hash of a single large file
2. Hash in chunks and report progress. Read the input in fixed chunks, update a progress counter after each chunk, and emit a progress event at least every thirty seconds with file name, bytes done, total bytes, and rate. Avoid loading whole files into memory, which also removes a common source of stalls. Verify: progress events appear throughout the run and the reported rate is consistent with the elapsed time
3. Checkpoint per file, and per chunk for very large files. Record each completed file with its resulting hash in the checkpoint as soon as it finishes, and for files above a size threshold, record enough state to resume or at least to prove progress, such as bytes hashed in the current session. On restart, skip files with a recorded hash for the same restore point. Verify: killing the worker mid-run and restarting does not rehash already completed files
4. Size the lease from the work, not a default. Estimate duration from total bytes and a measured hashing rate for this storage, set the orchestrator lease and kill threshold above that estimate with margin, and store the calculation in the run record. Recompute when the data size changes materially instead of keeping a fixed timeout inherited from quick jobs. Verify: the run record shows estimated duration, the rate used, and a lease above the estimate
5. Make the orchestrator judge progress, not silence. Where the orchestrator supports it, configure the failure rule so a run is only killed when heartbeats stop or progress stalls for a defined period, rather than on a single missed beat or a fixed wall clock for all sizes. A stall should be measured in bytes not advancing, which catches a truly hung worker that a heartbeat alone would miss. Verify: a slow but advancing run survives, while a deliberately hung test worker is still detected
6. Report duration honestly at close. Publish per-file and total hashing time, the data volume, and the achieved rate in the verification report, and flag files whose size makes them exceed the standard lease so scheduling can place them in a long window. Feed observed rates back into the estimator. Verify: the report lists the largest file, its hash duration, and whether it needed the extended lease

Why it happens

Hashing is a long, quiet, single threaded looking operation: the worker may produce no logs, no network calls, and no status changes for many minutes on a multi-terabyte archive. If the heartbeat is emitted by the same loop that is busy hashing, or only between files, the orchestrator sees silence and does exactly what it was configured to do. The fault is in the signalling design, not the data, and chunk progress plus an independent heartbeat makes the work visible at the cadence the orchestrator needs.

Variants and edge cases

Edge cases: on very slow or contended storage the hash rate can drop far below the estimate, so the progress based rule matters more than the lease calculation. A separate heartbeat can mask a genuinely hung main loop if it reports only alive, which is why it must carry the byte counter. Some orchestrators cap maximum run duration regardless of progress; for those, split verification by file ranges across runs with the checkpoint joining them.

orchestrator killed the checksum run as dead

Use when worker logs show steady progress up to the kill. Decouple the heartbeat and include bytes processed in it.

verification times out only on the biggest archive

Use when size predicts failure. Hash in chunks, checkpoint per file, and size the lease from measured rate.

heartbeat missed during long verify

Use when beats resume in later runs or smaller files. Emit heartbeats from an independent timer at a fixed interval.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4Mnyf38PqrLurOvBfYNuQ