Buildkite agent lost connection mid-job: how to diagnose

Not specified by the original author.

TL;DR

Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Agent lost connection

Steps

1. Check whether the agent is still alive and reporting:

buildkite-agent --version
ps aux | grep buildkite-agent | grep -v grep

Expected: the agent process is listed and running. If nothing is there, the agent died and took the job with it.

2. Read the agent log around the disconnect timestamp:

grep -i "lost connection\|heartbeat\|disconnect" [AGENT_LOG_PATH] | tail -30

Expected: log lines showing missed heartbeats, a ping timeout, or a clean shutdown. A clean shutdown points at host preemption; timeouts point at the network.

3. Check whether the host itself went away (spot preemption, autoscaling, OOM):

dmesg | grep -i "oom\|killed process" | tail -10
uptime

Expected: either an OOM kill naming the job process, or an uptime shorter than the job runtime, which proves the host was replaced mid-job.

4. Make the job retry automatically only when the loss was infrastructure, not a test failure:

steps:
  - label: "[JOB NAME]"
    command: [COMMAND]
    retry:
      automatic:
        - exit_status: -1
          limit: 2

Expected: exit status -1 (the code Buildkite assigns to lost agents) retries up to twice, while real failures still fail fast.

5. Confirm the agent fleet is healthy before the next run:

buildkite-agent meta-data get "buildkite:agent:version"

Expected: metadata returns, proving this agent can reach the Buildkite backend right now.

When to use

• A job ends with agent lost connection after starting normally
• Jobs die on specific agent hosts or at specific times of day (autoscaling windows)
• Self-hosted agents on spot or preemptible instances lose jobs under load

When not to use

• The job fails with a nonzero exit from your command; that is a step failure, not a lost agent
• The agent never picks up the job at all; that is a queue or tag mismatch
• The whole Buildkite status page shows an incident; that is upstream, not your agents

Why it happens

The agent holds a long-lived connection to the Buildkite backend and heartbeats through it while the job runs. If the agent process crashes, the host is preempted, the network partitions, or the job OOMs the box hard enough to wedge the agent, the heartbeats stop and the backend marks the job lost. The job itself was usually fine; its floor disappeared.

Edge cases

• Exit status -1 retries can mask a job that consistently OOMs its host; cap the retry limit and alert on repeated -1s per pipeline
• Corporate proxies that idle-timeout long connections kill agents that look idle between jobs; keepalive settings on the proxy matter as much as the agent config
• An agent running as PID 1 in a container that gets SIGTERM during deploys reports a clean disconnect; drain agents before rolling the fleet
• Clock skew on the agent host can break the heartbeat TLS session; NTP drift shows up as mysterious mass disconnects

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Thread query: Buildkite "agent lost connection" mid-job: how to diagnose

TL;DR: Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Not specified by the original author.

TL;DR

Check the agent logs and host metrics for networking, CPU, memory, and I/O pressure around the time heartbeats stopped. On the Elastic CI Stack for AWS with spot instances, abrupt termination also marks agents lost; use the log collector script to gather evidence.

Steps

1. Check the agent logs and host metrics for networking, CPU, memory, and I/O pressure around the time heartbeats stopped. On the Elastic CI Stack for AWS with spot instances, abrupt termination also marks agents lost; use the log collector script to gather evidence. Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout.

When to use

You are seeing this: Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout. Use this skill when you run into "Buildkite jobs fail with exit code -1: the agent stopped heartbeating".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout.

Source: Official Buildkite agent lifecycle docs: a -1 exit code means Buildkite lost contact with the agent. Agents send heartbeat updates after registering; if none arrive for three consecutive minutes, the agent is marked lost and gets no more jobs. Common causes are networking issues and CPU, memory, or I/O constraints on the agent host.

Verified resolution: Check the agent logs and host metrics for networking, CPU, memory, and I/O pressure around the time heartbeats stopped. On the Elastic CI Stack for AWS with spot instances, abrupt termination also marks agents lost; use the log collector script to gather evidence. Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout.

Source URL: https://github.com/buildkite/docs/blob/HEAD/pages/agent/lifecycle.md

Not specified by the original author.

TL;DR

Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.

The query

Jenkins agent disconnects mid-build: how to diagnose

Use this when

• Builds fail with agent went offline errors
• Specific agents disconnect repeatedly
• Disconnects correlate with load or time of day
• Agents on certain networks are unstable

Not for when

• Controller slowness or UI issues
• Builds that fail for code reasons while the agent stays up
• Setting up new agents (different task)

Steps

Step 1: Get the exact disconnect time and error

Find the build log's disconnect message and timestamp. Agent went offline, channel closed, ping timeout: each points a different direction. Correlate with the agent's own logs at the same timestamp. Expected output: a precise timestamp and error class for the disconnect.

Step 2: Check the agent host's resources at that time

Look at CPU, memory, and disk on the agent host around the disconnect. Agents get OOM-killed or wedged by disk-full conditions mid-build, and the controller just sees a dropped connection. Build agents doing heavy compiles are the usual victims. Expected output: either a resource spike matching the disconnect or resources ruled out.

Step 3: Check network path and timeouts

Verify connectivity between controller and agent: firewalls with idle timeouts killing long-lived connections is a classic. Compare against agents on networks that stay stable. Ping timeouts in the remoting layer are tunable if the network is merely slow rather than broken. Expected output: the network path characterized: stable, lossy, or firewalled with timeouts.

Step 4: Check clock skew

Large clock differences between controller and agent break authentication and session handling. Verify NTP sync on both sides. This one is quick to check and embarrassing to miss. Expected output: clocks in sync, or skew found and fixed.

Step 5: Look for external killers

Check whether anything reaps the agent process: autoscaling groups terminating instances, container orchestrators evicting pods, cron jobs cleaning processes, or security tools killing unknown binaries. The agent log ending abruptly (no shutdown message) suggests an external kill. Expected output: the killer identified (or ruled out), with protection added: termination protection, pod disruption budgets, or allowlisting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Thread query: Jenkins agent disconnects mid-build: how to diagnose

TL;DR: Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.