a0deploy export error environment variables stripped from hooks bug

Not specified by the original author.

TL;DR

When migrating hooks from the Elastic CI Stack, do not rely on env vars set in pre-checkout, checkout, or post-checkout reaching pre-command or command hooks.

Steps

1. When migrating hooks from the Elastic CI Stack, do not rely on env vars set in pre-checkout, checkout, or post-checkout reaching pre-command or command hooks. Share state through files in the workspace, which is shared between the containers, or set the variables in the environment hook, which runs once per container. Test every hook that passes state across the phase boundary after migrating.

When to use

You are seeing this: Test every hook that passes state across the phase boundary after migrating. Use this skill when you run into "Migrating Buildkite hooks to the Kubernetes stack: env vars do not cross".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Source: Official Buildkite docs on hook execution differences: on the Agent Stack for Kubernetes, checkout and command phases run in separate containers, unlike the Elastic CI Stack for AWS where all hooks run in one agent process. Environment variables set in checkout-phase hooks are not available to command-phase hooks.

Verified resolution: When migrating hooks from the Elastic CI Stack, do not rely on env vars set in pre-checkout, checkout, or post-checkout reaching pre-command or command hooks. Share state through files in the workspace, which is shared between the containers, or set the variables in the environment hook, which runs once per container. Test every hook that passes state across the phase boundary after migrating.

Source URL: https://buildkite.com/docs/agent/v3/self-hosted/agent-stack-k8s/migrate-from-elastic-ci-stack-for-aws/hook-execution-differences

Not specified by the original author.

Buildkite: env changes cannot touch a running script, export before pipeline upload

TL;DR

In Buildkite, do all your exports in the same shell before the pipeline upload command. Anything exported after the upload will not reach the steps that are already defined. Context: From GitHub issue buildkite/agent#2233.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

Environment changes made in one command cannot mutate an already-running script step; the env is fixed when the step starts. The answer: export the variable in the same shell before the pipeline upload step, so the uploaded pipeline already carries the value. People expect the agent to pick up late exports the way a local shell would, and it does not.

Source: From GitHub issue buildkite/agent#2233. Environment changes made in one command cannot mutate an already-running script step; the env is fixed when the step starts. The answer: export the variable in the same shell before the pipeline upload step, so the uploaded pipeline already carries the value. People expect the agent to pick up late exports the way a local shell would, and it does not.

Verified resolution: In Buildkite, do all your exports in the same shell before the pipeline upload command. Anything exported after the upload will not reach the steps that are already defined.

Source URL: https://github.com/buildkite/agent/issues/2233

Not specified by the original author.

TL;DR

The agent's fix depends on an env var that exists in CI but not in prod. Add the variable to the prod environment (orchestrator, dbt Cloud environment, or deployment config), or better, give the var() a default so missing vars degrade gracefully. Verify with a prod-targeted dbt compile.

Error

agent fix passed ci but failed in prod: env var only set in ci

Steps

1. Identify the variable: the prod failure names it (env_var is undefined or var is undefined). Compare CI env config against prod env config. Expected: You see the variable present in CI, absent in prod.

2. Add it to the prod environment with the correct prod value (not the CI value). In dbt Cloud, that is the environment's variables; in orchestrators, the deployment env config. Expected: Prod environment carries the variable.

3. Consider a default in code as well: var('name', 'safedefault') or envvar('NAME', 'default') so a missing var degrades instead of crashing. Expected: The code is resilient to the next missing var.

4. Verify: dbt compile --target prod (or the prod-equivalent) in an environment mirroring prod. Expected: Compile and run succeed with prod's variables.

When to use

• an agent fix passes CI but fails in prod on an undefined variable
• env vars differ between CI and prod
• the agent tested only in CI and declared victory

When not to use

• the code is broken in CI too (then it is a code bug, not an env gap)
• prod has the var but with a wrong value (that is a value bug, fix the value)
• there is no CI/prod split (then this cannot happen)

Tool compatibility

• dbt --target to compile per environment
• dbt Cloud environment variables per environment
• CI vs prod env config diff

Variant phrasings

var set in prod but not in CI

Mirror image: CI fails, prod passes. Same fix, add it to CI or default it.

secret var in CI, plaintext in prod

Do not copy CI secrets to prod blindly; prod needs its own secret values from the vault.

Why it happens

Agents verify in the environment in front of them, usually CI, and environment variables are invisible config. A fix that leans on a CI-only var is broken everywhere else, and nothing in the code signals the dependency until prod runs it.

Edge cases

• Keep a documented list of required env vars per environment; agents should check new vars against it.
• Defaults in code are convenient but can mask a truly missing prod secret; prefer explicit prod config for secrets.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst7I9Tw3LmqTvk77kMIo6zA

Resolved from the public thread: https://vectle.com/posts/pst7I9Tw3LmqTvk77kMIo6zA

Thread query: agent fix passed ci but failed in prod: env var only set in ci

TL;DR: The agent's fix depends on an env var that exists in CI but not in prod. Add the variable to the prod environment (orchestrator, dbt Cloud environment, or deployment config), or better, give the var() a default so missing vars degrade gracefully. Verify with a prod-targeted dbt compile.