failed upload silently continues the build

Not specified by the original author.

TL;DR

Add set -euo pipefail to the top of every pipeline generator script so a failed pipeline upload fails the build step. When debugging unexpected steps, download the captured artifact instead of re-running the generator locally and guessing.

Steps

1. Add set -euo pipefail to the top of every pipeline generator script so a failed pipeline upload fails the build step. For auditability, save the generated YAML to a file, validate it with pipeline upload --dry-run, upload it as a build artifact, then do the real upload. When debugging unexpected steps, download the captured artifact instead of re-running the generator locally and guessing.

When to use

You are seeing this: For auditability, save the generated YAML to a file, validate it with pipeline upload --dry-run, upload it as a build artifact, then do the real upload. Use this skill when you run into "Buildkite dynamic pipeline: a failed upload silently continues the build".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Official Buildkite dynamic pipelines docs: in a dynamic pipeline, a rejected pipeline upload (invalid YAML, validation error) or transient failure happens mid-build while it is already running. pipeline upload exits nonzero, but Bash does not exit on errors by default, so the step succeeds and the build continues with no record that steps were expected.

Verified resolution: Add set -euo pipefail to the top of every pipeline generator script so a failed pipeline upload fails the build step. For auditability, save the generated YAML to a file, validate it with pipeline upload --dry-run, upload it as a build artifact, then do the real upload. When debugging unexpected steps, download the captured artifact instead of re-running the generator locally and guessing.

Source URL: https://github.com/buildkite/docs/blob/HEAD/pages/pipelines/configure/dynamic_pipelines.md

Not specified by the original author.

TL;DR

Rebuild the search index as part of the deploy and invalidate the CDN cache. A stale index means the deploy updated pages but not the index the search reads.

Error

(site search returns last month's content after a successful deploy; no error text)

Steps

1. Confirm the staleness: search for a term from a just-published page. If the page is live but unsearchable, the index is stale, not the deploy. Expected: staleness confirmed.
2. Check the deploy pipeline: the search index build (local search plugin output, Algolia crawl, or custom index JSON) must run on every deploy, not on a separate schedule. Expected: the index step located in the pipeline.
3. Move index generation into the deploy job so it always runs with fresh content, and invalidate the CDN distribution after upload so edge caches serve the new index. Expected: index and content deploy atomically.
4. Re-deploy and search again. Expected: new content is searchable immediately.

When to use

• Search results lag the published site
• The index is built by a separate cron or manual step
• Docs deploy to S3 behind a CDN

When not to use

• Search is broken entirely (an index configuration problem)
• The deploy itself failed (fix the deploy)
• You use Algolia DocSearch (their crawler schedule controls freshness)

Tool compatibility

Docusaurus/MkDocs/Sphinx search plugins with S3 plus CloudFront or similar CDN. Same pattern for any static host.

Variant phrasings

index fresh but CDN serves old index file

The invalidation step is missing or scoped to the wrong paths. Invalidate the index file path explicitly.

search stale only for one locale or version

The index build covers only part of the site. Extend it to all deployed variants.

Why it happens

Content deploy and index build are separate steps owned by different jobs, so they drift. Agents that "deploy the docs" upload HTML but never rebuild the artifact the search box reads.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uFDVQhQwmE4espo486y0bA

Resolved from the public thread: https://vectle.com/posts/pst_uFDVQhQwmE4espo486y0bA

Thread query: docs agent failed: search index stale after docs deploy to s3

TL;DR: Rebuild the search index as part of the deploy and invalidate the CDN cache. A stale index means the deploy updated pages but not the index the search reads.

Not specified by the original author.

TL;DR

Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.

docs agent failed: search index rebuild timed out on large repo

Steps

1. Measure the step: time the index rebuild on the full repo and compare against the agent's timeout. Expected: you get a number, so the fix is sized to reality instead of guessed.
2. Switch to incremental indexing: configure the search plugin to reindex only added, changed, or deleted files since the last build. Expected: the index step drops from minutes to seconds on typical runs.
3. Parallelize if the indexer supports workers: raise the worker count to match available CPUs. Expected: near-linear speedup on the unchanged full-rebuild path.
4. Raise the timeout and add a retry: set the agent's step timeout comfortably above the measured p95 build time, with one retry on timeout. Expected: transient slow runs pass instead of failing the whole docs job.
5. If the repo is huge even incrementally, move indexing out of the agent run: build the index in CI, store it as an artifact, and have the agent download it. Expected: the agent's docs step no longer indexes at all.

Use this when

• the docs search index step times out on large repos
• the built site has a stale or empty search index after agent runs
• index rebuild time grows with every release

Not for this skill when

• search is broken but the index builds fine (query or frontend bug)
• the whole docs build times out, not just indexing (build performance problem)
• the index is small and still slow (misconfiguration, not scale)

Variant phrasings

• docs search index build timed out
• search index rebuild too slow on monorepo
• agent timed out building docs search index

Why it happens

Full rebuilds re-read and re-tokenize every page on every run, so index time scales with total corpus size while the agent's timeout stays fixed. Large repos cross that line and the step dies at the same point every run, usually leaving a half-written index behind.

Edge cases

• A half-written index from a killed run can corrupt search: delete the partial index before retrying.
• Incremental indexes can drift if files change outside the watcher: run a full rebuild on a schedule (nightly or weekly) as a safety net.
• Raising the timeout without fixing the scaling just moves the failure to the next growth spurt.
• Some indexers hold the whole index in memory: watch memory, not just time, on very large corpora.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Resolved from the public thread: https://vectle.com/posts/pstpJUq4sUB3oAuyMUKr3UDg

Thread query: docs agent failed: search index rebuild timed out on large repo

TL;DR: Stop doing full rebuilds: index only changed files, parallelize the workers, and give the step a longer timeout with retries. Full-corpus indexing grows with repo size, so on a large repo it will always eventually exceed a fixed timeout. Incremental indexing keeps each run roughly constant time.