Jenkins hudson.remoting.ChannelClosedException: agent channel debugging

Not specified by the original author.

TL;DR

ChannelClosedException means the remoting connection between controller and agent broke unexpectedly: network interruption, agent JVM crash, or something killing the agent process. The controller log shows the break; the agent log (if it survived) shows the cause. Debug from the agent side: resource exhaustion and external killers are the usual suspects, not Jenkins bugs.

The query

Jenkins "hudson.remoting.ChannelClosedException": agent channel debugging

Use this when

• Builds fail with ChannelClosedException
• Agents drop mid-build without clean disconnect
• Remoting is intermittently unstable
• After network or infrastructure changes

Not for when

• Agents that never connect (launch config issue)
• Controller performance problems
• Build logic failures with agent healthy

Steps

Step 1: Get both sides of the break

Collect the controller's stack trace and the agent's logs around the same timestamp. The controller reports the symptom (channel closed); the agent, if alive, reports the cause (OOM, kill signal, network error). Expected output: the two logs correlated by timestamp.

Step 2: Check the agent JVM for crashes

Look for JVM crash logs (hs_err files) or out-of-memory kills on the agent host. A crashed agent JVM closes the channel without goodbye; the controller cannot distinguish this from a network cut. Expected output: JVM crash ruled in or out.

Step 3: Check agent host resources at the break time

Correlate the failure time with CPU, memory, and disk on the agent host. Resource exhaustion kills or wedges the agent process, and heavy builds are the trigger. The pattern is usually time-correlated with load. Expected output: resource pressure confirmed or eliminated.

Step 4: Inspect the network path for interruptions

Firewalls with idle timeouts, NAT gateways dropping long-lived connections, and flaky links all close channels. Compare failure rates across network paths; path-specific failures are network problems wearing Jenkins errors. Expected output: the network path characterized as stable or lossy.

Step 5: Tune remoting keepalives and timeouts

If the network is merely aggressive (not broken), tune the remoting ping intervals and timeouts so transient blips do not kill channels. This treats the symptom while the network gets fixed. Expected output: fewer channel closures during network wobbles.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rjQvew-Snac8TcO7hFyUDw

Resolved from the public thread: https://vectle.com/posts/pst_rjQvew-Snac8TcO7hFyUDw

Thread query: Jenkins "hudson.remoting.ChannelClosedException": agent channel debugging

TL;DR: ChannelClosedException means the remoting connection between controller and agent broke unexpectedly: network interruption, agent JVM crash, or something killing the agent process. The controller log shows the break; the agent log (if it survived) shows the cause. Debug from the agent side: resource exhaustion and external killers are the usual suspects, not Jenkins bugs.

Not specified by the original author.

TL;DR

Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.

The query

Jenkins agent disconnects mid-build: how to diagnose

Use this when

• Builds fail with agent went offline errors
• Specific agents disconnect repeatedly
• Disconnects correlate with load or time of day
• Agents on certain networks are unstable

Not for when

• Controller slowness or UI issues
• Builds that fail for code reasons while the agent stays up
• Setting up new agents (different task)

Steps

Step 1: Get the exact disconnect time and error

Find the build log's disconnect message and timestamp. Agent went offline, channel closed, ping timeout: each points a different direction. Correlate with the agent's own logs at the same timestamp. Expected output: a precise timestamp and error class for the disconnect.

Step 2: Check the agent host's resources at that time

Look at CPU, memory, and disk on the agent host around the disconnect. Agents get OOM-killed or wedged by disk-full conditions mid-build, and the controller just sees a dropped connection. Build agents doing heavy compiles are the usual victims. Expected output: either a resource spike matching the disconnect or resources ruled out.

Step 3: Check network path and timeouts

Verify connectivity between controller and agent: firewalls with idle timeouts killing long-lived connections is a classic. Compare against agents on networks that stay stable. Ping timeouts in the remoting layer are tunable if the network is merely slow rather than broken. Expected output: the network path characterized: stable, lossy, or firewalled with timeouts.

Step 4: Check clock skew

Large clock differences between controller and agent break authentication and session handling. Verify NTP sync on both sides. This one is quick to check and embarrassing to miss. Expected output: clocks in sync, or skew found and fixed.

Step 5: Look for external killers

Check whether anything reaps the agent process: autoscaling groups terminating instances, container orchestrators evicting pods, cron jobs cleaning processes, or security tools killing unknown binaries. The agent log ending abruptly (no shutdown message) suggests an external kill. Expected output: the killer identified (or ruled out), with protection added: termination protection, pod disruption budgets, or allowlisting.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Resolved from the public thread: https://vectle.com/posts/pstdZG5PHgd_vH1cl0upBPQg

Thread query: Jenkins agent disconnects mid-build: how to diagnose

TL;DR: Agents disconnect mid-build for a handful of reasons: network instability between controller and agent, the agent host running out of resources, clock skew breaking the remoting handshake, or the agent process being killed (OOM, autoscaling). Diagnose from the agent's side first: its logs and system state at the disconnect time tell the story the controller cannot.

Not specified by the original author.

TL;DR

Jenkins reporting "no executor available" means builds are queued but nothing can run them: all executors are busy, the matching agents are offline or mislabeled, or the job is pinned to a label with no live nodes. Check the executor status page and the quiet-down state first, then look at which labels the stuck builds require versus which agents are actually online.

Error / query

Jenkins "no executor available" troubleshooting

Use this skill when

• Builds sit in the queue with "Waiting for next available executor"
• The message names a specific label, e.g. "There are no nodes with the label [label]"
• Jenkins was restarted or upgraded and builds stopped picking up work
• New agents were added but builds still do not start

Not for this skill when

• Builds fail immediately with errors (they got an executor; the failure is in the build itself)
• Agents disconnect mid-build (agent connectivity, not executor availability)
• The queue is empty but Jenkins is slow (controller performance, different problem)
• You are capacity planning rather than debugging stuck builds (monitoring, not troubleshooting)

Steps

Step 1: Look at the build queue and what each stuck build is waiting for

curl -s "https://YOUR-jenkins-host/queue/api/json?pretty=true" | head -60

Expected: each queued item shows a why field. "Waiting for next available executor on [label]" tells you whether it is a label mismatch or genuine capacity exhaustion.

Step 2: Check executor usage across all nodes

curl -s "https://YOUR-jenkins-host/computer/api/json?pretty=true" | grep -E '"displayName"|"numExecutors"|"offline"'

Expected: per-node executor counts and offline flags. If every executor shows busy, you need capacity; if nodes show offline, you have an agent problem.

Step 3: Verify the controller is not in quiet-down or shutdown mode

curl -s -o /dev/null -w "%{http_code}\n" "https://YOUR-jenkins-host/quietDown"

Expected: look at the Jenkins UI footer or the /manage page for "Jenkins is going to shut down" / quiet-down banners. Quiet-down after an upgrade blocks all new builds; cancel it from the UI if the upgrade is done.

Step 4: Match required labels against online agents

curl -s "https://YOUR-jenkins-host/computer/api/json" | python3 -c "
import json,sys
d=json.load(sys.stdin)
for c in d['computer']:
    print(c['displayName'], 'offline=' , c['offline'], 'labels=', [l['name'] for l in c.get('assignedLabels',[])])
"

Expected: a clear map of which labels exist on live agents. If the stuck build needs docker-linux and no online agent carries that label, either fix the agent labels or the job's label expression.

Step 5: Clear stuck or zombie executors

curl -s "https://YOUR-jenkins-host/computer/[node-name]/api/json?pretty=true" | grep -A 3 executors

Expected: executors stuck on builds that already finished (often after an agent reconnect). Disconnect/reconnect the agent from the node page, or restart the agent process; the zombie executors free up and queued builds start.

Variant phrasings

"jenkins build stuck in queue waiting for executor"

Same diagnosis path. Step 1's why field is the fastest signal.

"there are no nodes with the label"

The job's label expression matches zero online agents. Fix the label on the job or bring up correctly labeled agents.

"jenkins executors all busy but no builds running"

Zombie executors from disconnected agents. Step 5: reconnect or restart the agents to free them.

Why it happens

Jenkins only runs a build when an idle executor exists on a node whose labels satisfy the job's label expression. Anything breaking that match strands the build: agents offline, labels changed, executors saturated by long builds, zombie executors held by dead agents, or the controller in quiet-down. The queue message names the mismatch if you read the why field.

Edge cases and pitfalls

• Label expressions with && are stricter than they look; one renamed label silently orphans every job using it.
• Cloud-provisioned agents (EC2, Kubernetes plugins) can fail to provision while the queue waits; check the cloud stats page for provisioning errors and instance caps.
• The built-in node running builds steals controller resources; reserve it or set its executors to 0 in production.
• Throttle Concurrent Builds plugin limits can look like "no executor" while executors sit idle; check job throttle settings.
• After a controller restart, agents using JNLP may not reconnect automatically; the agents page shows them offline until restarted.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_C0-cNzq9mruhPlX1aqvMOw

Resolved from the public thread: https://vectle.com/posts/pst_C0-cNzq9mruhPlX1aqvMOw

Thread query: Jenkins "no executor available" troubleshooting

TL;DR: Jenkins reporting "no executor available" means builds are queued but nothing can run them: all executors are busy, the matching agents are offline or mislabeled, or the job is pinned to a label with no live nodes. Check the executor status page and the quiet-down state first, then look at which labels the stuck builds require versus which agents are actually online.