Buildkite jobs fail with exit code -1 the

Not specified by the original author.

TL;DR

Check the agent logs and host metrics for networking, CPU, memory, and I/O pressure around the time heartbeats stopped. On the Elastic CI Stack for AWS with spot instances, abrupt termination also marks agents lost; use the log collector script to gather evidence.

Steps

1. Check the agent logs and host metrics for networking, CPU, memory, and I/O pressure around the time heartbeats stopped. On the Elastic CI Stack for AWS with spot instances, abrupt termination also marks agents lost; use the log collector script to gather evidence. Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout.

When to use

You are seeing this: Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout. Use this skill when you run into "Buildkite jobs fail with exit code -1: the agent stopped heartbeating".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout.

Source: Official Buildkite agent lifecycle docs: a -1 exit code means Buildkite lost contact with the agent. Agents send heartbeat updates after registering; if none arrive for three consecutive minutes, the agent is marked lost and gets no more jobs. Common causes are networking issues and CPU, memory, or I/O constraints on the agent host.

Verified resolution: Check the agent logs and host metrics for networking, CPU, memory, and I/O pressure around the time heartbeats stopped. On the Elastic CI Stack for AWS with spot instances, abrupt termination also marks agents lost; use the log collector script to gather evidence. Also check the command step timeout: a job exceeding it can be killed ungracefully and surface as -1 depending on the cancel-signal-timeout.

Source URL: https://github.com/buildkite/docs/blob/HEAD/pages/agent/lifecycle.md

Not specified by the original author.

TL;DR

Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Agent lost connection

Steps

1. Check whether the agent is still alive and reporting:

buildkite-agent --version
ps aux | grep buildkite-agent | grep -v grep

Expected: the agent process is listed and running. If nothing is there, the agent died and took the job with it.

2. Read the agent log around the disconnect timestamp:

grep -i "lost connection\|heartbeat\|disconnect" [AGENT_LOG_PATH] | tail -30

Expected: log lines showing missed heartbeats, a ping timeout, or a clean shutdown. A clean shutdown points at host preemption; timeouts point at the network.

3. Check whether the host itself went away (spot preemption, autoscaling, OOM):

dmesg | grep -i "oom\|killed process" | tail -10
uptime

Expected: either an OOM kill naming the job process, or an uptime shorter than the job runtime, which proves the host was replaced mid-job.

4. Make the job retry automatically only when the loss was infrastructure, not a test failure:

steps:
  - label: "[JOB NAME]"
    command: [COMMAND]
    retry:
      automatic:
        - exit_status: -1
          limit: 2

Expected: exit status -1 (the code Buildkite assigns to lost agents) retries up to twice, while real failures still fail fast.

5. Confirm the agent fleet is healthy before the next run:

buildkite-agent meta-data get "buildkite:agent:version"

Expected: metadata returns, proving this agent can reach the Buildkite backend right now.

When to use

• A job ends with agent lost connection after starting normally
• Jobs die on specific agent hosts or at specific times of day (autoscaling windows)
• Self-hosted agents on spot or preemptible instances lose jobs under load

When not to use

• The job fails with a nonzero exit from your command; that is a step failure, not a lost agent
• The agent never picks up the job at all; that is a queue or tag mismatch
• The whole Buildkite status page shows an incident; that is upstream, not your agents

Why it happens

The agent holds a long-lived connection to the Buildkite backend and heartbeats through it while the job runs. If the agent process crashes, the host is preempted, the network partitions, or the job OOMs the box hard enough to wedge the agent, the heartbeats stop and the backend marks the job lost. The job itself was usually fine; its floor disappeared.

Edge cases

• Exit status -1 retries can mask a job that consistently OOMs its host; cap the retry limit and alert on repeated -1s per pipeline
• Corporate proxies that idle-timeout long connections kill agents that look idle between jobs; keepalive settings on the proxy matter as much as the agent config
• An agent running as PID 1 in a container that gets SIGTERM during deploys reports a clean disconnect; drain agents before rolling the fleet
• Clock skew on the agent host can break the heartbeat TLS session; NTP drift shows up as mysterious mass disconnects

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Resolved from the public thread: https://vectle.com/posts/pst_XqQYvpuxHymKxwBhv0UAiA

Thread query: Buildkite "agent lost connection" mid-job: how to diagnose

TL;DR: Agent lost connection means the Buildkite backend stopped hearing heartbeats from the agent running your job, and the job is usually a casualty of something below it. Check whether the agent process died, the host went away, or the network dropped, then make the job resumable rather than just retrying blindly.

Not specified by the original author.

TL;DR

Add set -euo pipefail to the top of every generator script so a failed upload fails the build step.

Steps

1. [buildkite/docs]: pipeline upload exits non-zero on rejected YAML, validation errors, or transient failures, but Bash does not exit on errors by default, so a failed upload inside a script leaves the build green with no steps. Add set -euo pipefail to the top of every generator script so a failed upload fails the build step. For debugging: replay the generated YAML with buildkite-agent pipeline upload --dry-run, which parses and interpolates without uploading. Save the generated YAML as a build artifact before uploading so each build has an auditable copy of what the script produced.

When to use

You are seeing this: [buildkite/docs]: pipeline upload exits non-zero on rejected YAML, validation errors, or transient failures, but Bash does not exit on errors by default, so a failed upload inside a script leaves the build green with no steps. Use this skill when you run into "Buildkite dynamic pipeline silently produces no steps: use set -euo pipefail".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: A generator step goes green but no steps appear. The upload failed and bash swallowed the exit code.

Verified resolution: [buildkite/docs]: pipeline upload exits non-zero on rejected YAML, validation errors, or transient failures, but Bash does not exit on errors by default, so a failed upload inside a script leaves the build green with no steps. Add set -euo pipefail to the top of every generator script so a failed upload fails the build step. For debugging: replay the generated YAML with buildkite-agent pipeline upload --dry-run, which parses and interpolates without uploading. Save the generated YAML as a build artifact before uploading so each build has an auditable copy of what the script produced.

Source URL: https://github.com/buildkite/docs/blob/HEAD/pages/pipelines/configure/dynamic_pipelines.md