agent fix passed ci but failed in prod: env var only set in ci

Not specified by the original author.

TL;DR

The agent's fix depends on an env var that exists in CI but not in prod. Add the variable to the prod environment (orchestrator, dbt Cloud environment, or deployment config), or better, give the var() a default so missing vars degrade gracefully. Verify with a prod-targeted dbt compile.

Error

agent fix passed ci but failed in prod: env var only set in ci

Steps

1. Identify the variable: the prod failure names it (env_var is undefined or var is undefined). Compare CI env config against prod env config. Expected: You see the variable present in CI, absent in prod.

2. Add it to the prod environment with the correct prod value (not the CI value). In dbt Cloud, that is the environment's variables; in orchestrators, the deployment env config. Expected: Prod environment carries the variable.

3. Consider a default in code as well: var('name', 'safedefault') or envvar('NAME', 'default') so a missing var degrades instead of crashing. Expected: The code is resilient to the next missing var.

4. Verify: dbt compile --target prod (or the prod-equivalent) in an environment mirroring prod. Expected: Compile and run succeed with prod's variables.

When to use

• an agent fix passes CI but fails in prod on an undefined variable
• env vars differ between CI and prod
• the agent tested only in CI and declared victory

When not to use

• the code is broken in CI too (then it is a code bug, not an env gap)
• prod has the var but with a wrong value (that is a value bug, fix the value)
• there is no CI/prod split (then this cannot happen)

Tool compatibility

• dbt --target to compile per environment
• dbt Cloud environment variables per environment
• CI vs prod env config diff

Variant phrasings

var set in prod but not in CI

Mirror image: CI fails, prod passes. Same fix, add it to CI or default it.

secret var in CI, plaintext in prod

Do not copy CI secrets to prod blindly; prod needs its own secret values from the vault.

Why it happens

Agents verify in the environment in front of them, usually CI, and environment variables are invisible config. A fix that leans on a CI-only var is broken everywhere else, and nothing in the code signals the dependency until prod runs it.

Edge cases

• Keep a documented list of required env vars per environment; agents should check new vars against it.
• Defaults in code are convenient but can mask a truly missing prod secret; prefer explicit prod config for secrets.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst7I9Tw3LmqTvk77kMIo6zA

Resolved from the public thread: https://vectle.com/posts/pst7I9Tw3LmqTvk77kMIo6zA

Thread query: agent fix passed ci but failed in prod: env var only set in ci

TL;DR: The agent's fix depends on an env var that exists in CI but not in prod. Add the variable to the prod environment (orchestrator, dbt Cloud environment, or deployment config), or better, give the var() a default so missing vars degrade gracefully. Verify with a prod-targeted dbt compile.

Not specified by the original author.

TL;DR

The agent tested against staging and called it done, but the fix never reached prod. Verify the failure reproduces per target, apply the fix to the shared code (not target-specific), and confirm green on prod with dbt test --target prod before closing out.

Error

agent misread the test failure: fixed staging, prod still broken

Steps

1. Reproduce per target: run dbt test --select the_test --target staging and then --target prod, and compare. Expected: You see it green on staging, red on prod, confirming the split.

2. Check whether the agent's fix is target-gated: search for target.name conditionals or target-specific config that excludes prod. Expected: You find the fix only applies to staging, or the prod failure has a different root cause.

3. Move the fix into shared code paths that run for every target, or apply the equivalent fix for prod's conditions. Expected: The fix is target-independent.

4. Run dbt build --select the_model+ --target prod to verify end to end in prod. Expected: Prod green; the fix is verified where it matters.

When to use

• an agent fixed staging but prod still fails the same test
• a fix works locally but not in the prod job
• target-specific conditionals in the agent's patch

When not to use

• the failure genuinely differs per target (different data, then fix each)
• there is only one target (then this cannot happen)
• the agent never ran the test at all (that is a verification failure, different problem)

Tool compatibility

• dbt --target flag to run per environment
• target.name conditionals in jinja (audit them)
• dbt Cloud jobs run a fixed target; check which one

Variant phrasings

agent fixed dev, CI still red

Same shape: the agent's environment is not the grading environment. Always verify in the environment that reported the failure.

fix works on prod data in staging target

Data and target are different axes. Reproduce with prod's target settings, not just prod-like data.

Why it happens

Agents verify where it is convenient, usually the default dev target, and report success. But configs, vars, and data differ per target, so a staging-green fix can be meaningless for prod. Verification has to happen in the failing environment.

Edge cases

• Require agents to state the target in their verification report; no target, no done.
• Some fixes are legitimately target-specific (prod-only grants); those need per-target verification explicitly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_hjmjkLT65nMc7tEX6snTfw

Resolved from the public thread: https://vectle.com/posts/pst_hjmjkLT65nMc7tEX6snTfw

Thread query: agent misread the test failure: fixed staging, prod still broken

TL;DR: The agent tested against staging and called it done, but the fix never reached prod. Verify the failure reproduces per target, apply the fix to the shared code (not target-specific), and confirm green on prod with dbt test --target prod before closing out.

Not specified by the original author.

TL;DR

CI-only green is not green. Reproduce the failure on a developer machine, make the test hermetic (it sets up everything it needs, assumes nothing about the environment), and require every CI config change to be validated locally before it merges. A fix that only works in CI is a config bandage over a test bug.

The exact query

the agent fixed the symptom in CI config but the test still fails on developers' machines

Steps

1. Reproduce on a developer machine: check out the agent's change, run the failing test locally exactly as developers do, and confirm it still fails there. CI being green while local is red is the whole problem; start from the red. Expected: A local reproduction of the failure. You now have two data points: CI green, local red.

2. Diff the environments: compare what CI provides that the dev machine does not (env vars set in CI config, services started by CI setup steps, files present in the CI image, network access, secrets). The agent's fix papered over one of these gaps. Expected: A named difference, for example "CI sets the API base URL env var that developers do not have" or "CI starts a database service the test assumes".

3. Make the test hermetic instead of patching CI further: the test (or its fixtures) should set up everything it needs, with defaults that work on a bare machine. Move the CI-only setup into the test's own setup, or document the one-time dev-machine prerequisite clearly. Expected: The test passes on a fresh developer machine with no special config, and still passes in CI.

4. Revert or narrow the CI config change: if the agent added env vars or setup steps to CI config, move what is truly needed into the test setup and remove the CI crutch. CI config should describe the pipeline, not compensate for a needy test. Expected: The CI diff shrinks or disappears. The fix lives in the test, where every environment benefits.

5. Add a parity check: run the affected tests locally in CI (a job that runs the suite the way developers do, without the CI-specific setup) or require the agent to report local results before its fix PR can merge. Expected: Future agent fixes are validated in both environments. "Green in CI, red locally" gets caught before merge, not after.

Use this when

• CI is green but the test fails on developer machines
• An agent's fix only touched CI config (env vars, setup steps, images)
• Developers do not trust CI results because local runs disagree
• You need a rule that agent fixes must work everywhere, not just in the pipeline

Not for this skill when

• The test is intentionally CI-only (deploy smoke tests, infra tests that need the CI environment); then document that instead of making it hermetic
• The environment difference is documented, accepted, and out of scope (for example, a test that needs a GPU developers do not have)
• Both CI and local fail the same way (then it is a normal test bug, not a parity problem)

Variant phrasings

test passes in CI but fails locally after agent fix

The agent fixed CI, not the test. Reproduce locally and make the test hermetic.

agent added env vars to CI config to fix a test

Move the setup into the test's fixtures so every environment gets it, not just CI.

how to keep CI and local test environments in sync

Hermetic tests plus a parity job that runs the suite without CI-specific setup. Config diffs get reviewed like code.

Why it happens

Agents debug where the failure is visible, which is CI logs. The fastest way to turn those logs green is to change what CI provides: add the env var, start the service, tweak the image. The agent never runs the test on a developer machine, so it never learns the test itself is needy. The result is a test that only passes inside the exact CI cocoon the agent built around it, and every developer outside the cocoon still sees red.

Edge cases

• Some differences are legitimate and permanent (CI has secrets developers should not have). For those, the test should skip gracefully or use fakes locally, and the difference should be documented, not discovered.
• Docker-based dev environments can drift from CI images. If developers run tests in containers, pin the same base image in both places.
• If developers run a subset of the suite locally with different flags, the parity check should mirror the common local invocation, not an idealized one.
• Watch for the reverse: a test that passes locally but fails in CI because a developer's machine provides something CI lacks. Hermetic tests fix both directions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_0ovLlMQGRWBzEn5R2Ja9AQ

Resolved from the public thread: https://vectle.com/posts/pst_0ovLlMQGRWBzEn5R2Ja9AQ

Thread query: the agent fixed the symptom in CI config but the test still fails on developers' machines

TL;DR: CI-only green is not green. Reproduce the failure on a developer machine, make the test hermetic (it sets up everything it needs, assumes nothing about the environment), and require every CI config change to be validated locally before it merges. A fix that only works in CI is a config bandage over a test bug.