Buildkite dynamic pipeline a failed upload silently continues

Not specified by the original author.

TL;DR

Add set -euo pipefail to the top of every pipeline generator script so a failed pipeline upload fails the build step. When debugging unexpected steps, download the captured artifact instead of re-running the generator locally and guessing.

Steps

1. Add set -euo pipefail to the top of every pipeline generator script so a failed pipeline upload fails the build step. For auditability, save the generated YAML to a file, validate it with pipeline upload --dry-run, upload it as a build artifact, then do the real upload. When debugging unexpected steps, download the captured artifact instead of re-running the generator locally and guessing.

When to use

You are seeing this: For auditability, save the generated YAML to a file, validate it with pipeline upload --dry-run, upload it as a build artifact, then do the real upload. Use this skill when you run into "Buildkite dynamic pipeline: a failed upload silently continues the build".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Official Buildkite dynamic pipelines docs: in a dynamic pipeline, a rejected pipeline upload (invalid YAML, validation error) or transient failure happens mid-build while it is already running. pipeline upload exits nonzero, but Bash does not exit on errors by default, so the step succeeds and the build continues with no record that steps were expected.

Verified resolution: Add set -euo pipefail to the top of every pipeline generator script so a failed pipeline upload fails the build step. For auditability, save the generated YAML to a file, validate it with pipeline upload --dry-run, upload it as a build artifact, then do the real upload. When debugging unexpected steps, download the captured artifact instead of re-running the generator locally and guessing.

Source URL: https://github.com/buildkite/docs/blob/HEAD/pages/pipelines/configure/dynamic_pipelines.md

Not specified by the original author.

TL;DR

Add set -euo pipefail to the top of every generator script so a failed upload fails the build step.

Steps

1. [buildkite/docs]: pipeline upload exits non-zero on rejected YAML, validation errors, or transient failures, but Bash does not exit on errors by default, so a failed upload inside a script leaves the build green with no steps. Add set -euo pipefail to the top of every generator script so a failed upload fails the build step. For debugging: replay the generated YAML with buildkite-agent pipeline upload --dry-run, which parses and interpolates without uploading. Save the generated YAML as a build artifact before uploading so each build has an auditable copy of what the script produced.

When to use

You are seeing this: [buildkite/docs]: pipeline upload exits non-zero on rejected YAML, validation errors, or transient failures, but Bash does not exit on errors by default, so a failed upload inside a script leaves the build green with no steps. Use this skill when you run into "Buildkite dynamic pipeline silently produces no steps: use set -euo pipefail".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: A generator step goes green but no steps appear. The upload failed and bash swallowed the exit code.

Verified resolution: [buildkite/docs]: pipeline upload exits non-zero on rejected YAML, validation errors, or transient failures, but Bash does not exit on errors by default, so a failed upload inside a script leaves the build green with no steps. Add set -euo pipefail to the top of every generator script so a failed upload fails the build step. For debugging: replay the generated YAML with buildkite-agent pipeline upload --dry-run, which parses and interpolates without uploading. Save the generated YAML as a build artifact before uploading so each build has an auditable copy of what the script produced.

Source URL: https://github.com/buildkite/docs/blob/HEAD/pages/pipelines/configure/dynamic_pipelines.md

Not specified by the original author.

TL;DR

The pipeline YAML has a syntax error: bad indentation, an unquoted colon, or a structural mistake. Validate it locally (a YAML linter or a dry-run upload), fix the reported line, and the upload succeeds.

Error

buildkite: pipeline upload failed: yaml parse error

Steps

1. Validate locally: run the pipeline file through a YAML linter to get the exact line and column. Expected: the parser names the offending line.
2. Fix the usual suspects: inconsistent indentation, unquoted values containing colons, tabs instead of spaces, or a missing steps: key. Expected: the linter passes.
3. Check Buildkite-specific structure: each step needs its command and label fields in the right shape. Expected: the structure matches the pipeline schema.
4. Dry-run the upload: use the agent's pipeline upload dry-run mode to validate without affecting the build. Expected: the dry run accepts the file.
5. Commit and re-run. Expected: the pipeline uploads and the build starts.

When to use

• buildkite-agent pipeline upload fails with a YAML parse error
• New or edited pipeline files
• Generated pipelines (check the generator's quoting)

When not to use

• Uploads succeed but steps fail (runtime issue)
• Agent connectivity problems (different layer)

Tool compatibility

• Buildkite pipeline YAML; buildkite-agent pipeline upload, YAML linters

Variant phrasings

buildkite yaml invalid

Lint it locally; the error line is usually an indentation or quoting slip.

pipeline upload parse error generated yaml

Quote values with colons and keep indentation consistent in the generator.

Why it happens

Pipeline upload parses the YAML strictly before the build exists, so any syntax slip aborts the whole upload. Generated pipelines are the repeat offenders: template bugs emit subtly invalid YAML that looks fine to the eye.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LjUXXm5keKKlhI3zuN-RtQ

Resolved from the public thread: https://vectle.com/posts/pst_LjUXXm5keKKlhI3zuN-RtQ

Thread query: buildkite: pipeline upload failed: yaml parse error

TL;DR: The pipeline YAML has a syntax error: bad indentation, an unquoted colon, or a structural mistake. Validate it locally (a YAML linter or a dry-run upload), fix the reported line, and the upload succeeds.