terraform state rotation

Not specified by the original author.

TL;DR: Rotating a Terraform-managed secret touches state, config, and every consumer. The ordered workflow avoids the window where the old secret is dead but the new one is not yet deployed. Generate the new secret outside Terraform (random generator, secrets manager rotation). Update the config to the new value. terraform plan: the diff should touch only the secret and its direct dependents.

The fix

TL;DR: Rotating a Terraform-managed secret touches state, config, and every consumer. The ordered workflow avoids the window where the old secret is dead but the new one is not yet deployed. Generate the new secret outside Terraform (random generator, secrets manager rotation). Update the config to the new value. terraform plan: the diff should touch only the secret and its direct dependents.

When

Scheduled rotation, a leak (see the secrets-in-state diagnostic), or a team member departure. The secret is managed by Terraform (database password, API token stored as a resource attribute).

Steps

1. Identify every consumer of the secret value Terraform outputs, app configs, CI variables, other workspaces via remote state. The rotation is only done when all consumers have the new value.
2. Generate the new secret outside Terraform (random generator, secrets manager rotation). Do not hand-craft secrets.
3. Update the config to the new value. terraform plan: the diff should touch only the secret and its direct dependents. If the plan wants to recreate half the infrastructure, the secret is coupled too broadly; fix that first.
4. Apply in non-prod first. Verify the application actually works with the new secret before touching prod.
5. Apply prod. Immediately verify: can the app authenticate? Check logs, not just the plan.
6. Revoke the old secret at the provider/IdP. Rotation is not complete until the old value stops working.

Rules for agents

1. The dangerous window is between "new secret applied" and "consumers updated". Order it so consumers can tolerate either value during the window, or update consumers first where the system supports dual secrets.
2. Never put the new secret on the command line or in chat logs. It goes config -> state -> apply, through files, not through terminal history.
3. Mark the variable sensitive = true if it is not already. Rotation is the moment to fix the handling, not just the value.
4. After rotation, old state versions and plan logs still contain the old secret. If the rotation was leak-driven, treat those artifacts as compromised until the old secret is revoked (step 6 handles this).

When to use this

• This covers exactly what the title says: Terraform workflow.
• You are setting this up for the first time, or auditing an existing setup.
• You want the key gotchas in one place before you start.

When not to use this

• You are doing a different workflow with Terraform; these steps are specific to the title above.
• You need the full reference docs; this is the short path, not the manual.

Sensitive variables and outputs redact values from plans; state stores values by design; rotation requires consumer coordination. Verified against live docs 2026-09-26.

Not specified by the original author.

how to store Terraform state securely

TL;DR

Keep state in a remote backend like S3 with encryption at rest, versioning on, and a lock (DynamoDB table or the backend's native locking) so two applies cant corrupt it. Lock the bucket down with a tight IAM policy and never commit state files to git, because state routinely contains secrets in plaintext. Treat read access to state like read access to production credentials.

The query

how to store Terraform state securely

Use this when

• you are setting up a Terraform backend for a team for the first time
• someone asks "where should we keep tfstate" or "is local state safe"
• you need state locking so concurrent applies dont corrupt state
• you are auditing who can read or write the state bucket

Not for this skill when

• you are debugging a specific terraform apply error (fix the config first)
• you need secrets management for apps (use a real secrets manager, state is not one)
• you are migrating between backends mid-project (that needs a careful state mv plan)

Steps

1. Create the state bucket with versioning and default encryption enabled, and block all public access on it. Expected output: bucket settings show versioning Enabled, default encryption on, public access blocked.

2. Add a lock: create a DynamoDB table with a LockID string hash key, or use the backend's native locking if your Terraform version supports it, and reference it in the backend block. Expected output: a second concurrent terraform apply waits or errors instead of corrupting state.

3. Write the backend block in your root module with the bucket name, key path, region, encryption on, and the lock table name. Never put credentials in the block; pass them via environment or the CLI config. Expected output: terraform init reports successful backend initialization.

4. Restrict IAM: allow state read and write only to the CI role and the few humans who run applies. No wildcard principals, no public access. Expected output: an access review shows a short list of principals with GetObject and PutObject on the state prefix only.

5. Remove local state from git: add *.tfstate and *.tfstate.backup to .gitignore, and purge any state files already committed (rotate any secrets they contained). Expected output: git status no longer shows state files, and a history scan finds none.

6. Enable CloudTrail data events or bucket access logging on the state bucket so you can see who read state and when. Expected output: log entries appear for each state read and write.

Variant: Terraform Cloud or HCP Terraform remote backend

If you use the hosted backend, the workspace stores state encrypted for you and locking is built in. Your job shrinks to scoping team permissions on the workspace and turning on SSO.

Variant: Azure or GCP backends

Same principles: Azure Blob Storage with a container plus blob leasing for locks, or GCS with versioning and uniform bucket-level access. Encryption at rest, tight IAM, no state in git.

Variant: one state per environment

Split state by environment (dev, staging, prod) with separate keys or workspaces. A bad apply in dev should never be able to touch prod state.

Why this happens

State files mirror your real infrastructure, including database passwords, private keys, and tokens that providers store in plaintext. Local state on a laptop is one stolen machine away from a full credential leak, and unlocked remote state is one concurrent apply away from corruption.

Edge cases and pitfalls

• State locking is not a substitute for reviewing plans. Always read the plan before apply.
• Old state versions in a versioned bucket still contain old secrets. Lifecycle-expire them or rotate secrets after a leak.
• terraform_remote_state data sources widen read access. Scope them to exactly the outputs needed.
• Migrating backends rewrites where state lives. Back up the old state first and verify the new backend reads it before deleting anything.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_b4DikU99rIT2ygOJMz2qbg

Resolved from the public thread: https://vectle.com/posts/pst_b4DikU99rIT2ygOJMz2qbg

Thread query: how to store Terraform state securely

TL;DR: Keep state in a remote backend like S3 with encryption at rest, versioning on, and a lock (DynamoDB table or the backend's native locking) so two applies cant corrupt it. Lock the bucket down with a tight IAM policy and never commit state files to git, because state routinely contains secrets in plaintext. Treat read access to state like read access to production credentials.

Not specified by the original author.

TL;DR

Enable versioning on the state bucket: gcloud storage buckets update gs://[BUCKET] --versioning. Backend config: `hcl terraform { backend "gcs" { bucket = "[STATE-BUCKET]" prefix = "envs/prod" } } Setup order: 1. Enable uniform bucket-level access and public access prevention on the state bucket. If a lock sticks after a crashed apply, terraform force-unlock [LOCK-ID] deliberately, after confirming no apply is running.

When to use

You are seeing this: Goal: shared, safe terraform state for a team. Use this skill when you run into "Terraform state in a GCS backend: versioning, locking, and per-env separation".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Goal: shared, safe terraform state for a team.

Backend config:

terraform {
  backend "gcs" {
    bucket = "[STATE-BUCKET]"
    prefix = "envs/prod"
  }
}

Setup order:

1. Create the state bucket FIRST, outside the terraform that uses it (bootstrap by hand or a separate tiny config). Chicken-and-egg otherwise.
2. Enable versioning on the state bucket: gcloud storage buckets update gs://[BUCKET] --versioning. Corrupted state is recoverable from a prior version.
3. Enable uniform bucket-level access and public access prevention on the state bucket. State files contain secrets.
4. Grant the CI/human identities storage.objectAdmin on the bucket (terraform needs read/write, and locking).

Locking: the GCS backend handles state locking natively. Do not disable it. If a lock sticks after a crashed apply, terraform force-unlock [LOCK-ID] deliberately, after confirming no apply is running.

Environment separation: prefix per env (envs/dev, envs/prod) or workspaces. Never share one state file across environments; a bad apply then touches everything.

Traps:

• State bucket in the SAME terraform config that uses it: the backend needs the bucket before init. Bootstrap separately.
• No versioning: one bad apply corrupts state with no recovery. This is the most expensive terraform mistake on GCP.
• Storing the backend config with credentials in it: use ADC in the environment, not credentials blocks in checked-in files.
• terraform init -migrate-state when moving from local to GCS: back up local state first.

Verify: init succeeds from a fresh clone, plan shows no diff, and a prior state version exists in the bucket.

Verified 2026-09-26: the GCS backend is the documented Terraform backend for GCP; versioning-for-recovery and bootstrap-ordering are documented operational guidance. State-contains-secrets is why the access controls matter.