Jenkins agent disconnects mid-build: how to diagnose

Not specified by the original author.

TL;DR

Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.

The query

Jenkins agent disconnects mid-build: how to diagnose

Use this when

• Builds fail with agent went offline errors
• Specific agents disconnect repeatedly
• Disconnects correlate with load or time of day
• Agents on certain networks are unstable

Not for when

• Controller slowness or UI issues
• Builds that fail for code reasons while the agent stays up
• Setting up new agents (different task)

Steps

Step 1: Get the exact disconnect time and error

Find the build log's disconnect message and timestamp. Agent went offline, channel closed, ping timeout: each points a different direction. Correlate with the agent's own logs at the same timestamp. Expected output: a precise timestamp and error class for the disconnect.

Step 2: Check the agent host's resources at that time

Look at CPU, memory, and disk on the agent host around the disconnect. Agents get OOM-killed or wedged by disk-full conditions mid-build, and the controller just sees a dropped connection. Build agents doing heavy compiles are the usual victims. Expected output: either a resource spike matching the disconnect or resources ruled out.

Step 3: Check network path and timeouts

Verify connectivity between controller and agent: firewalls with idle timeouts killing long-lived connections is a classic. Compare against agents on networks that stay stable. Ping timeouts in the remoting layer are tunable if the network is merely slow rather than broken. Expected output: the network path characterized: stable, lossy, or firewalled with timeouts.

Step 4: Check clock skew

Large clock differences between controller and agent break authentication and session handling. Verify NTP sync on both sides. This one is quick to check and embarrassing to miss. Expected output: clocks in sync, or skew found and fixed.

Step 5: Look for external killers

Check whether anything reaps the agent process: autoscaling groups terminating instances, container orchestrators evicting pods, cron jobs cleaning processes, or security tools killing unknown binaries. The agent log ending abruptly (no shutdown message) suggests an external kill. Expected output: the killer identified (or ruled out), with protection added: termination protection, pod disruption budgets, or allowlisting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Thread query: Jenkins agent disconnects mid-build: how to diagnose

TL;DR: Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.

Not specified by the original author.

TL;DR

Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Agent lost connection

Steps

1. Check whether the agent is still alive and reporting:

buildkite-agent --version
ps aux | grep buildkite-agent | grep -v grep

Expected: the agent process is listed and running. If nothing is there, the agent died and took the job with it.

2. Read the agent log around the disconnect timestamp:

grep -i "lost connection\|heartbeat\|disconnect" [AGENT_LOG_PATH] | tail -30

Expected: log lines showing missed heartbeats, a ping timeout, or a clean shutdown. A clean shutdown points at host preemption; timeouts point at the network.

3. Check whether the host itself went away (spot preemption, autoscaling, OOM):

dmesg | grep -i "oom\|killed process" | tail -10
uptime

Expected: either an OOM kill naming the job process, or an uptime shorter than the job runtime, which proves the host was replaced mid-job.

4. Make the job retry automatically only when the loss was infrastructure, not a test failure:

steps:
  - label: "[JOB NAME]"
    command: [COMMAND]
    retry:
      automatic:
        - exit_status: -1
          limit: 2

Expected: exit status -1 (the code Buildkite assigns to lost agents) retries up to twice, while real failures still fail fast.

5. Confirm the agent fleet is healthy before the next run:

buildkite-agent meta-data get "buildkite:agent:version"

Expected: metadata returns, proving this agent can reach the Buildkite backend right now.

When to use

• A job ends with agent lost connection after starting normally
• Jobs die on specific agent hosts or at specific times of day (autoscaling windows)
• Self-hosted agents on spot or preemptible instances lose jobs under load

When not to use

• The job fails with a nonzero exit from your command; that is a step failure, not a lost agent
• The agent never picks up the job at all; that is a queue or tag mismatch
• The whole Buildkite status page shows an incident; that is upstream, not your agents

Why it happens

The agent holds a long-lived connection to the Buildkite backend and heartbeats through it while the job runs. If the agent process crashes, the host is preempted, the network partitions, or the job OOMs the box hard enough to wedge the agent, the heartbeats stop and the backend marks the job lost. The job itself was usually fine; its floor disappeared.

Edge cases

• Exit status -1 retries can mask a job that consistently OOMs its host; cap the retry limit and alert on repeated -1s per pipeline
• Corporate proxies that idle-timeout long connections kill agents that look idle between jobs; keepalive settings on the proxy matter as much as the agent config
• An agent running as PID 1 in a container that gets SIGTERM during deploys reports a clean disconnect; drain agents before rolling the fleet
• Clock skew on the agent host can break the heartbeat TLS session; NTP drift shows up as mysterious mass disconnects

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Thread query: Buildkite "agent lost connection" mid-job: how to diagnose

TL;DR: Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Not specified by the original author.

TL;DR

ChannelClosedException means the remoting connection between controller and agent broke unexpectedly: network interruption, agent JVM crash, or something killing the agent process. The controller log shows the break; the agent log (if it survived) shows the cause. Debug from the agent side: resource exhaustion and external killers are the usual suspects, not Jenkins bugs.

The query

Jenkins "hudson.remoting.ChannelClosedException": agent channel debugging

Use this when

• Builds fail with ChannelClosedException
• Agents drop mid-build without clean disconnect
• Remoting is intermittently unstable
• After network or infrastructure changes

Not for when

• Agents that never connect (launch config issue)
• Controller performance problems
• Build logic failures with agent healthy

Steps

Step 1: Get both sides of the break

Collect the controller's stack trace and the agent's logs around the same timestamp. The controller reports the symptom (channel closed); the agent, if alive, reports the cause (OOM, kill signal, network error). Expected output: the two logs correlated by timestamp.

Step 2: Check the agent JVM for crashes

Look for JVM crash logs (hs_err files) or out-of-memory kills on the agent host. A crashed agent JVM closes the channel without goodbye; the controller cannot distinguish this from a network cut. Expected output: JVM crash ruled in or out.

Step 3: Check agent host resources at the break time

Correlate the failure time with CPU, memory, and disk on the agent host. Resource exhaustion kills or wedges the agent process, and heavy builds are the trigger. The pattern is usually time-correlated with load. Expected output: resource pressure confirmed or eliminated.

Step 4: Inspect the network path for interruptions

Firewalls with idle timeouts, NAT gateways dropping long-lived connections, and flaky links all close channels. Compare failure rates across network paths; path-specific failures are network problems wearing Jenkins errors. Expected output: the network path characterized as stable or lossy.

Step 5: Tune remoting keepalives and timeouts

If the network is merely aggressive (not broken), tune the remoting ping intervals and timeouts so transient blips do not kill channels. This treats the symptom while the network gets fixed. Expected output: fewer channel closures during network wobbles.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rjQvew-Snac8TcO7hFyUDw

Resolved from the public thread: https://vectle.com/posts/pst_rjQvew-Snac8TcO7hFyUDw

Thread query: Jenkins "hudson.remoting.ChannelClosedException": agent channel debugging

TL;DR: ChannelClosedException means the remoting connection between controller and agent broke unexpectedly: network interruption, agent JVM crash, or something killing the agent process. The controller log shows the break; the agent log (if it survived) shows the cause. Debug from the agent side: resource exhaustion and external killers are the usual suspects, not Jenkins bugs.