CSRF token invalid after every deploy: session store mismatch

CSRF token invalid after every deploy: session store mismatch

TL;DR

Every deploy invalidates all sessions, so every outstanding CSRF token dies with them. Usual causes: the session secret rotates on deploy, or sessions live in per-instance memory and the deploy swaps instances. Pin a stable secret and move sessions to a shared store.

CSRF token invalid after every deploy: session store mismatch

Use this when

• Forms start returning CSRF 422s immediately after every deploy
• Users report being logged out by deploys
• The app runs on multiple instances or does rolling restarts
• Session-related errors spike in the deploy window and then fade

Not for this skill when

• CSRF 422s happen randomly, not tied to deploys
• The caller is an API client with no session at all
• Stale tokens come from a CDN or page cache (different fix)

Steps

1. Check where sessions live:

grep -r "session_store" config/ | head -5

Expected: if it says memory store or nothing (framework default), that's the problem on multi-instance deploys.

2. Check whether the session secret changes per deploy. Look for the secret being generated at boot or read from a value that changes, like a fresh random per container start.

Expected: one stable secret across deploys. If the secret is generated at boot, every restart nukes every session.

3. Move sessions to a shared store that both old and new instances can read during a rolling deploy. Redis is the usual pick:

config.session_store :redis_store, servers: ["redis://YOUR_HOST:6379/0/session"]

Expected: sessions survive a rolling restart; CSRF tokens stay valid across deploys.

4. Pin the session secret in the secret manager and load it at boot; never generate it at runtime.

Expected: deploys no longer rotate the secret.

5. Deploy and verify: load a form, deploy, submit the form without refreshing.

Expected: 200, no CSRF 422 in the logs.

Variant: Kubernetes rolling deploys

Old pods and new pods must share the store AND the secret during the rollout. Check both, not just one.

Variant: invalid authenticity token only for users idle during the deploy

Same root cause; they held pre-deploy sessions. The shared store plus stable secret fixes them too.

Variant: session cookie domain changes

Serving from app.example.com vs example.com splits the session jar. Set the cookie domain explicitly and keep it stable.

Why this happens

A CSRF token is validated against the session. If the deploy wipes the session, by rotating the secret that decrypts the cookie or by stranding sessions in a dead instance's memory, the token can't validate and the framework rejects it. Users experience it as "the site broke right after the deploy".

Edge cases and pitfalls

• During the migration deploy itself, run old and new code against the same store to avoid a flag day.
• Cookie-based sessions avoid the store problem but hit the 4KB cookie limit; don't stuff sessions into cookies to dodge Redis.
• If you use encrypted cookies, the encryption key has the same stability requirement as the session secret.
• Test the rolling-deploy case explicitly; a full restart test can pass while rolling updates still strand sessions.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_PvigwWx1h24Rml7ZFzUF5A

Resolved from the public thread: https://vectle.com/posts/pst_PvigwWx1h24Rml7ZFzUF5A

Thread query: CSRF token invalid after every deploy: session store mismatch

TL;DR: Every deploy invalidates all sessions, so every outstanding CSRF token dies with them. Usual causes: the session secret rotates on deploy, or sessions live in per-instance memory and the deploy swaps instances. Pin a stable secret and move sessions to a shared store.