Diagnosing a stale Terraform state lock: "Error acquiring the state lock" from a dead process

Diagnosing a stale Terraform state lock

Symptom

Error: Error acquiring the state lock, and the Lock Info Who/Created point at a process that should be finished: a CI job that ended, a laptop that rebooted, a terminal that was closed.

Cause

The lock was never released because the holder died without cleanup: SIGKILLed CI job, OOM-killed runner, closed laptop lid mid-apply. The lock record in DynamoDB (or the S3 lockfile) outlives the process.

Confirmation

1. Read the Lock Info ID. Note the exact lock ID string.
2. Prove the holder is dead: check the CI job status for that run ID, ps on the machine named in Who, or the runner's job history. A finished/failed job with no running terraform process means stale.
3. Check whether the dead run half-applied: you cannot know yet, and that is fine. Do not touch state until the lock is cleared and you can plan.

Fix

1. terraform force-unlock [lock-id] with the exact ID from the error. It asks for confirmation; this is the one prompt you answer deliberately.
2. Immediately run terraform plan (not apply). Read the whole plan.

Verification

1. The plan tells you what the dead run did: creates/updates that already happened show as no-change or as drift to reconcile; unfinished work shows as pending changes.
2. If the plan is clean or only shows expected changes, the state is consistent. Proceed normally.

3) If the plan shows half-created resources or confusing diffs, reconcile before applying: import what exists, remove what is broken, then apply.

1. Fix the root cause: CI jobs that get SIGKILLed need graceful shutdown or lock timeouts; runners need enough memory to not OOM mid-apply.