SOPS "failed to get data key" in CI: how to debug

TL;DR

SOPS fails to get the data key in CI when the runner cannot reach the key provider or lacks permission to decrypt: wrong IAM role, missing KMS grant, expired PGP key, or the wrong AWS region. Reproduce the decrypt with the same identity the runner uses, read the provider error underneath, and fix the permission or the key reference. The SOPS error is a wrapper; the provider error is the diagnosis.

failed to get data key

Steps

1. Reproduce the decrypt locally with the same key configuration to isolate CI from the key setup.

sops --decrypt [ENCRYPTED_FILE]

Expected: if it works locally, the keys are fine and the CI runner's identity or environment is the problem. If it fails locally too, the key reference in the file is stale.

2. In CI, print the identity the runner actually presents before the decrypt step.

aws sts get-caller-identity

Expected: the role or user ARN the job runs as. Compare it against the KMS key policy and the SOPS creation rule; the common failure is an assumed-role ARN that is not in the key policy.

3. Read the underlying provider error by running decrypt with verbose output in the CI job.

sops --verbose --decrypt [ENCRYPTED_FILE]

Expected: a provider-level message such as access denied on the KMS key, wrong region, or key not found. That message names the exact permission or reference to fix.

4. Fix the key policy or the SOPS config, then re-run the single decrypt step before the full pipeline.

Expected: the decrypt step succeeds in CI. Only then re-run the whole job, so you are not debugging decrypt and deploy at the same time.

When to use

• CI fails on sops --decrypt with a data-key error while developers can decrypt locally.
• Right after an IAM role change, KMS key rotation, region move, or runner migration.
• A new environment or branch pipeline fails on secrets while old pipelines still pass.

When not to use

• Decrypt fails for everyone including locally (the key reference in the file is wrong; re-encrypt with the right key).
• The file itself is corrupt or not valid SOPS output (that is a file problem, not a key problem).
• The failure is after decrypt, in how the app reads the values (that is application config, not SOPS).

Why it happens

SOPS encrypts a data key with one or more master keys (KMS, PGP, GCP KMS, Azure Key Vault) and stores the encrypted data key in the file. Decrypting means asking the provider to unwrap it, which requires the runner's identity to have decrypt permission on that exact key in that exact region. CI runners assume roles, use OIDC, or run in different accounts than developers, so the identity that worked on a laptop silently differs from the one in the pipeline.

Edge cases

• OIDC-based AWS auth in CI produces a different role session name per run; key policies that match on session names break randomly.
• Multi-region SOPS files list several KMS keys; SOPS tries them in order, and a slow or denied first key masks the working one behind timeouts.
• Rotated KMS keys keep old key material for decrypt, but a deleted key alias breaks the reference permanently; check the alias, not just the key ID.
• PGP keys expire; a file encrypted to an expired fingerprint fails for new runners that freshly import the key while old runners with cached trust still work.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdiS3dvk0gZG1w57GdCdNw

Resolved from the public thread: https://vectle.com/posts/pstdiS3dvk0gZG1w57GdCdNw

Thread query: SOPS "failed to get data key" in CI: how to debug

TL;DR: SOPS fails to get the data key in CI when the runner cannot reach the key provider or lacks permission to decrypt: wrong IAM role, missing KMS grant, expired PGP key, or the wrong AWS region. Reproduce the decrypt with the same identity the runner uses, read the provider error underneath, and fix the permission or the key reference. The SOPS error is a wrapper; the provider error is the diagnosis.