Buildkite "agent lost connection" mid-job: how to diagnose

TL;DR

Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Agent lost connection

Steps

1. Check whether the agent is still alive and reporting:

buildkite-agent --version
ps aux | grep buildkite-agent | grep -v grep

Expected: the agent process is listed and running. If nothing is there, the agent died and took the job with it.

2. Read the agent log around the disconnect timestamp:

grep -i "lost connection\|heartbeat\|disconnect" [AGENT_LOG_PATH] | tail -30

Expected: log lines showing missed heartbeats, a ping timeout, or a clean shutdown. A clean shutdown points at host preemption; timeouts point at the network.

3. Check whether the host itself went away (spot preemption, autoscaling, OOM):

dmesg | grep -i "oom\|killed process" | tail -10
uptime

Expected: either an OOM kill naming the job process, or an uptime shorter than the job runtime, which proves the host was replaced mid-job.

4. Make the job retry automatically only when the loss was infrastructure, not a test failure:

steps:
  - label: "[JOB NAME]"
    command: [COMMAND]
    retry:
      automatic:
        - exit_status: -1
          limit: 2

Expected: exit status -1 (the code Buildkite assigns to lost agents) retries up to twice, while real failures still fail fast.

5. Confirm the agent fleet is healthy before the next run:

buildkite-agent meta-data get "buildkite:agent:version"

Expected: metadata returns, proving this agent can reach the Buildkite backend right now.

When to use

• A job ends with agent lost connection after starting normally
• Jobs die on specific agent hosts or at specific times of day (autoscaling windows)
• Self-hosted agents on spot or preemptible instances lose jobs under load

When not to use

• The job fails with a nonzero exit from your command; that is a step failure, not a lost agent
• The agent never picks up the job at all; that is a queue or tag mismatch
• The whole Buildkite status page shows an incident; that is upstream, not your agents

Why it happens

The agent holds a long-lived connection to the Buildkite backend and heartbeats through it while the job runs. If the agent process crashes, the host is preempted, the network partitions, or the job OOMs the box hard enough to wedge the agent, the heartbeats stop and the backend marks the job lost. The job itself was usually fine; its floor disappeared.

Edge cases

• Exit status -1 retries can mask a job that consistently OOMs its host; cap the retry limit and alert on repeated -1s per pipeline
• Corporate proxies that idle-timeout long connections kill agents that look idle between jobs; keepalive settings on the proxy matter as much as the agent config
• An agent running as PID 1 in a container that gets SIGTERM during deploys reports a clean disconnect; drain agents before rolling the fleet
• Clock skew on the agent host can break the heartbeat TLS session; NTP drift shows up as mysterious mass disconnects

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA