Jenkins agent disconnects mid-build: how to diagnose

TL;DR

Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.

The query

Jenkins agent disconnects mid-build: how to diagnose

Use this when

• Builds fail with agent went offline errors
• Specific agents disconnect repeatedly
• Disconnects correlate with load or time of day
• Agents on certain networks are unstable

Not for when

• Controller slowness or UI issues
• Builds that fail for code reasons while the agent stays up
• Setting up new agents (different task)

Steps

Step 1: Get the exact disconnect time and error

Find the build log's disconnect message and timestamp. Agent went offline, channel closed, ping timeout: each points a different direction. Correlate with the agent's own logs at the same timestamp. Expected output: a precise timestamp and error class for the disconnect.

Step 2: Check the agent host's resources at that time

Look at CPU, memory, and disk on the agent host around the disconnect. Agents get OOM-killed or wedged by disk-full conditions mid-build, and the controller just sees a dropped connection. Build agents doing heavy compiles are the usual victims. Expected output: either a resource spike matching the disconnect or resources ruled out.

Step 3: Check network path and timeouts

Verify connectivity between controller and agent: firewalls with idle timeouts killing long-lived connections is a classic. Compare against agents on networks that stay stable. Ping timeouts in the remoting layer are tunable if the network is merely slow rather than broken. Expected output: the network path characterized: stable, lossy, or firewalled with timeouts.

Step 4: Check clock skew

Large clock differences between controller and agent break authentication and session handling. Verify NTP sync on both sides. This one is quick to check and embarrassing to miss. Expected output: clocks in sync, or skew found and fixed.

Step 5: Look for external killers

Check whether anything reaps the agent process: autoscaling groups terminating instances, container orchestrators evicting pods, cron jobs cleaning processes, or security tools killing unknown binaries. The agent log ending abruptly (no shutdown message) suggests an external kill. Expected output: the killer identified (or ruled out), with protection added: termination protection, pod disruption budgets, or allowlisting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Thread query: Jenkins agent disconnects mid-build: how to diagnose

TL;DR: Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.