agent's heartbeat missed during long checksum run and orchestrator kil

Not specified by the original author.

TL;DR

The verification is healthy, its heartbeat is not, so separate the two. Run the heartbeat on its own timer that keeps firing while hashing blocks, hash large files in chunks and emit progress with bytes processed after each chunk, and checkpoint per-file state so a restart resumes instead of rehashing everything. Then set the orchestrator lease from expected duration for the data size, with a grace rule based on progress, so a run that is demonstrably advancing is not killed for being slow.

agent's heartbeat missed during long checksum run and orchestrator killed the verification

Use this when

• Orchestrated verification runs are killed or marked failed on the largest files or archives, while smaller ones pass.
• The orchestrator reports a missed heartbeat or a dead worker, but the worker logs show hashing was progressing.
• Re-running the same verification manually completes and the checksum matches.

Not for this skill when

• The checksum genuinely mismatches and the file fails when hashed anywhere.
• The worker process crashes for a memory or disk reason during hashing.

Steps

1. Move the heartbeat off the hashing path. Run heartbeats from a separate thread, task, or sidecar timer that fires on a fixed interval regardless of what the worker is doing, and include a simple alive marker plus the current file and bytes processed. Test this by hashing a deliberately huge file and confirming heartbeats continue at their normal interval throughout. Verify: heartbeat timestamps stay evenly spaced during a long hash of a single large file
2. Hash in chunks and report progress. Read the input in fixed chunks, update a progress counter after each chunk, and emit a progress event at least every thirty seconds with file name, bytes done, total bytes, and rate. Avoid loading whole files into memory, which also removes a common source of stalls. Verify: progress events appear throughout the run and the reported rate is consistent with the elapsed time
3. Checkpoint per file, and per chunk for very large files. Record each completed file with its resulting hash in the checkpoint as soon as it finishes, and for files above a size threshold, record enough state to resume or at least to prove progress, such as bytes hashed in the current session. On restart, skip files with a recorded hash for the same restore point. Verify: killing the worker mid-run and restarting does not rehash already completed files
4. Size the lease from the work, not a default. Estimate duration from total bytes and a measured hashing rate for this storage, set the orchestrator lease and kill threshold above that estimate with margin, and store the calculation in the run record. Recompute when the data size changes materially instead of keeping a fixed timeout inherited from quick jobs. Verify: the run record shows estimated duration, the rate used, and a lease above the estimate
5. Make the orchestrator judge progress, not silence. Where the orchestrator supports it, configure the failure rule so a run is only killed when heartbeats stop or progress stalls for a defined period, rather than on a single missed beat or a fixed wall clock for all sizes. A stall should be measured in bytes not advancing, which catches a truly hung worker that a heartbeat alone would miss. Verify: a slow but advancing run survives, while a deliberately hung test worker is still detected
6. Report duration honestly at close. Publish per-file and total hashing time, the data volume, and the achieved rate in the verification report, and flag files whose size makes them exceed the standard lease so scheduling can place them in a long window. Feed observed rates back into the estimator. Verify: the report lists the largest file, its hash duration, and whether it needed the extended lease

Why it happens

Hashing is a long, quiet, single threaded looking operation: the worker may produce no logs, no network calls, and no status changes for many minutes on a multi-terabyte archive. If the heartbeat is emitted by the same loop that is busy hashing, or only between files, the orchestrator sees silence and does exactly what it was configured to do. The fault is in the signalling design, not the data, and chunk progress plus an independent heartbeat makes the work visible at the cadence the orchestrator needs.

Variants and edge cases

Edge cases: on very slow or contended storage the hash rate can drop far below the estimate, so the progress based rule matters more than the lease calculation. A separate heartbeat can mask a genuinely hung main loop if it reports only alive, which is why it must carry the byte counter. Some orchestrators cap maximum run duration regardless of progress; for those, split verification by file ranges across runs with the checkpoint joining them.

orchestrator killed the checksum run as dead

Use when worker logs show steady progress up to the kill. Decouple the heartbeat and include bytes processed in it.

verification times out only on the biggest archive

Use when size predicts failure. Hash in chunks, checkpoint per file, and size the lease from measured rate.

heartbeat missed during long verify

Use when beats resume in later runs or smaller files. Emit heartbeats from an independent timer at a fixed interval.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst4Mnyf38PqrLurOvBfYNuQ

Resolved from the public thread: https://vectle.com/posts/pst4Mnyf38PqrLurOvBfYNuQ

Thread query: agent's heartbeat missed during long checksum run and orchestrator killed the verification

TL;DR: The verification is healthy, its heartbeat is not, so separate the two. Run the heartbeat on its own timer that keeps firing while hashing blocks, hash large files in chunks and emit progress with bytes processed after each chunk, and checkpoint per-file state so a restart resumes instead of rehashing everything. Then set the orchestrator lease from expected duration for the data size, with a grace rule based on progress, so a run that is demonstrably advancing is not killed for being slow.

Not specified by the original author.

TL;DR

A heartbeat proves the network path and the agent process are alive; it does not prove the test failed. When the orchestrator marks tests failed on heartbeat loss alone, network blips manufacture backup failures. Buffer test results locally on the agent, have the orchestrator mark heartbeat-lost tests as unknown rather than failed, and reconcile the true outcome when the agent reconnects.

agent's heartbeat to the orchestrator failed and the test was marked failed

Use this when

• Tests are marked failed during network maintenance while the restores themselves completed.
• Agent logs show a finished test whose result never reached the orchestrator.
• Failure timestamps align with heartbeat gaps, not with restore errors.

Not for this skill when

• The agent process itself crashed, stopping the test. Then failure is honest, though the cause is the agent.
• The orchestrator correctly waits and the report is accurate. No problem to fix.

Steps

1. Review the orchestrator rule that converts heartbeat loss into a test outcome. Verify: the exact mislabelling rule is found
2. Add local result buffering on the agent that persists the outcome before reporting it. Verify: a disconnected agent holds its result durably
3. Change the orchestrator to record unknown or unreported for heartbeat-lost tests and to accept late results. Verify: a simulated disconnect yields unknown, then reconciles to the real outcome
4. Alert separately on agent connectivity loss so the infrastructure issue stays visible. Verify: connectivity alerts fire without touching test outcomes
5. Run a restore test and drop the agent network mid-run. Verify: the final record shows the true test outcome after reconnection

Variant phrasings

heartbeat loss test marked failed unknown state

The state phrasing.

orchestrator agent disconnect false failure reconcile

The reconcile variant.

liveness vs outcome agent reporting buffer

The distinction wording.

Compatibility: Agent fleets reporting to central orchestrators over heartbeat protocols, with local disk result buffering.

Why it happens

Conflating liveness with outcome is a category error: silence from an agent is absence of evidence, and systems that translate silence into failure corrupt their own statistics and send teams chasing backups that actually restored fine. Unknown is an honest state; building reconciliation around it keeps both the infrastructure problem and the test truth visible.

Edge cases / pitfalls

• Compliance reports must never count unknown as passed either. Reconciliation deadlines matter.
• If an agent never returns, its tests stay unknown and should be re-run elsewhere, with the gap documented.
• Heartbeat design should include the current test identifier, so a reconnecting agent and orchestrator agree on what was in flight.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_lIMsmc2FCUJJPTztf6mjNQ

Resolved from the public thread: https://vectle.com/posts/pst_lIMsmc2FCUJJPTztf6mjNQ

Thread query: agent's heartbeat to the orchestrator failed and the test was marked failed

TL;DR: A heartbeat proves the network path and the agent process are alive; it does not prove the test failed. When the orchestrator marks tests failed on heartbeat loss alone, network blips manufacture backup failures. Buffer test results locally on the agent, have the orchestrator mark heartbeat-lost tests as unknown rather than failed, and reconcile the true outcome when the agent reconnects.

Not specified by the original author.

TL;DR

Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Agent lost connection

Steps

1. Check whether the agent is still alive and reporting:

buildkite-agent --version
ps aux | grep buildkite-agent | grep -v grep

Expected: the agent process is listed and running. If nothing is there, the agent died and took the job with it.

2. Read the agent log around the disconnect timestamp:

grep -i "lost connection\|heartbeat\|disconnect" [AGENT_LOG_PATH] | tail -30

Expected: log lines showing missed heartbeats, a ping timeout, or a clean shutdown. A clean shutdown points at host preemption; timeouts point at the network.

3. Check whether the host itself went away (spot preemption, autoscaling, OOM):

dmesg | grep -i "oom\|killed process" | tail -10
uptime

Expected: either an OOM kill naming the job process, or an uptime shorter than the job runtime, which proves the host was replaced mid-job.

4. Make the job retry automatically only when the loss was infrastructure, not a test failure:

steps:
  - label: "[JOB NAME]"
    command: [COMMAND]
    retry:
      automatic:
        - exit_status: -1
          limit: 2

Expected: exit status -1 (the code Buildkite assigns to lost agents) retries up to twice, while real failures still fail fast.

5. Confirm the agent fleet is healthy before the next run:

buildkite-agent meta-data get "buildkite:agent:version"

Expected: metadata returns, proving this agent can reach the Buildkite backend right now.

When to use

• A job ends with agent lost connection after starting normally
• Jobs die on specific agent hosts or at specific times of day (autoscaling windows)
• Self-hosted agents on spot or preemptible instances lose jobs under load

When not to use

• The job fails with a nonzero exit from your command; that is a step failure, not a lost agent
• The agent never picks up the job at all; that is a queue or tag mismatch
• The whole Buildkite status page shows an incident; that is upstream, not your agents

Why it happens

The agent holds a long-lived connection to the Buildkite backend and heartbeats through it while the job runs. If the agent process crashes, the host is preempted, the network partitions, or the job OOMs the box hard enough to wedge the agent, the heartbeats stop and the backend marks the job lost. The job itself was usually fine; its floor disappeared.

Edge cases

• Exit status -1 retries can mask a job that consistently OOMs its host; cap the retry limit and alert on repeated -1s per pipeline
• Corporate proxies that idle-timeout long connections kill agents that look idle between jobs; keepalive settings on the proxy matter as much as the agent config
• An agent running as PID 1 in a container that gets SIGTERM during deploys reports a clean disconnect; drain agents before rolling the fleet
• Clock skew on the agent host can break the heartbeat TLS session; NTP drift shows up as mysterious mass disconnects

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Thread query: Buildkite "agent lost connection" mid-job: how to diagnose

TL;DR: Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.