agent failed over to the preview environment but preview secrets were never set, worker returned 500s

TL;DR

Production secrets do not automatically exist in preview. The agent failed over to the preview environment, the code read secrets that were never set there, and every request 500d. Set the preview secrets (with preview-safe values), verify the worker reads them, and make secret setup part of every environment's checklist.

The query

agent failed over to the preview environment but preview secrets were never set, worker returned 500s

Use this when

• Preview worker returns 500s on secret reads.
• The same code works in production.
• Preview was set up recently or secrets were added after preview setup.

Not for

• Production secret failures (different environment).
• Preview failures unrelated to secrets (debug those directly).
• Secrets set in preview but with wrong values (value problem).

Steps

Step 1: Confirm the missing secrets

Check which secret reads fail in preview. The 500s correlate with secret access.

Expected result: the exact secret names missing in preview. Not a code bug; a config gap.

Step 2: Set preview-safe secret values

Create preview versions of each secret. Use test credentials and sandbox keys, never production values.

Expected result: preview has a complete secret set with safe values. Production secrets never leak into preview.

Step 3: Verify the worker reads them

Exercise the secret-dependent paths in preview.

Expected result: 500s gone. The worker behaves like production minus real credentials.

Step 4: Add secret setup to the environment checklist

Every new environment (preview, staging, per-developer) gets its secret set as part of provisioning, before any traffic.

Expected result: the gap cannot recur. Environments are born complete.

Step 5: Make the agent check secrets before failover

Before failing over, the agent verifies the target environment's secret set is complete.

Expected result: failover to an unprepared environment is blocked with a clear message instead of producing mysterious 500s.

Why it happens

Secrets are per-environment by design, but humans think of them as 'the app's secrets' (singular). Preview environments get created for code and routes while secrets are forgotten. The failure is delayed until the first failover, which is exactly when you can least afford a config surprise.

Edge cases

• Some secrets have no safe preview equivalent (production-only webhooks). Stub them with documented fake values and note the behavioral difference.
• Rotating a production secret does not rotate preview. Track both sets in the rotation runbook.
• The agent that failed over may have cached the 500s as 'preview is broken'. Clear that conclusion after the fix.
• Secret names must match across environments even when values differ. A typo'd name in preview is the same bug wearing a different hat.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_p5yVQ9u9vn2HuwTdFoI72Q