agent fix passed ci but failed in prod: env var only set in ci

TL;DR

The agent's fix depends on an env var that exists in CI but not in prod. Add the variable to the prod environment (orchestrator, dbt Cloud environment, or deployment config), or better, give the var() a default so missing vars degrade gracefully. Verify with a prod-targeted dbt compile.

Error

agent fix passed ci but failed in prod: env var only set in ci

Steps

1. Identify the variable: the prod failure names it (env_var is undefined or var is undefined). Compare CI env config against prod env config. Expected: You see the variable present in CI, absent in prod.

2. Add it to the prod environment with the correct prod value (not the CI value). In dbt Cloud, that is the environment's variables; in orchestrators, the deployment env config. Expected: Prod environment carries the variable.

3. Consider a default in code as well: var('name', 'safedefault') or envvar('NAME', 'default') so a missing var degrades instead of crashing. Expected: The code is resilient to the next missing var.

4. Verify: dbt compile --target prod (or the prod-equivalent) in an environment mirroring prod. Expected: Compile and run succeed with prod's variables.

When to use

• an agent fix passes CI but fails in prod on an undefined variable
• env vars differ between CI and prod
• the agent tested only in CI and declared victory

When not to use

• the code is broken in CI too (then it is a code bug, not an env gap)
• prod has the var but with a wrong value (that is a value bug, fix the value)
• there is no CI/prod split (then this cannot happen)

Tool compatibility

• dbt --target to compile per environment
• dbt Cloud environment variables per environment
• CI vs prod env config diff

Variant phrasings

var set in prod but not in CI

Mirror image: CI fails, prod passes. Same fix, add it to CI or default it.

secret var in CI, plaintext in prod

Do not copy CI secrets to prod blindly; prod needs its own secret values from the vault.

Why it happens

Agents verify in the environment in front of them, usually CI, and environment variables are invisible config. A fix that leans on a CI-only var is broken everywhere else, and nothing in the code signals the dependency until prod runs it.

Edge cases

• Keep a documented list of required env vars per environment; agents should check new vars against it.
• Defaults in code are convenient but can mask a truly missing prod secret; prefer explicit prod config for secrets.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst7I9Tw3LmqTvk77kMIo6zA