chef encrypted data bag failed to decrypt: key mismatch

Not specified by the original author.

TL;DR

A Chef node decrypts data bags with its local data-bag secret; if that secret differs from the one used at encryption time, decryption fails. Sync the correct secret file to the node (or re-encrypt the bag with the node's key) and the run converges.

Error

chef encrypted data bag failed to decrypt: key mismatch

Steps

1. Locate the secret on the node: check the encrypted_data_bag_secret path in the client config and confirm the file exists. Expected: a real file at the configured path.
2. Compare with the encryption side: verify which secret was used to encrypt the data bag item (check your workstation or CI encryption step). Expected: you can state whether the two secrets match.
3. Sync the correct secret to the node via your configuration management or secret distribution (never paste it into chat or tickets). Expected: the node's secret file matches the encryption secret byte for byte.
4. If the secret legitimately rotated, re-encrypt the data bag items with the new secret and upload them to the Chef server. Expected: the server holds items encrypted with the current secret.
5. Re-run chef-client on the node. Expected: the data bag decrypts and the run completes.

When to use

• chef-client fails decrypting an encrypted data bag with a key mismatch
• After rotating the data-bag secret without re-encrypting
• New nodes that received a different secret than existing ones

When not to use

• Chef server connectivity failures (network, not crypto)
• Cookbook syntax errors (unrelated to the data bag)

Tool compatibility

• Chef Infra Client, encrypted data bags (knife data bag), encrypted_data_bag_secret

Variant phrasings

data bag secret rotated but items not re-encrypted

Re-encrypt every item with the new secret and upload; old items stay unreadable to the new key.

new chef node cannot decrypt data bag

The node got a different secret file; distribute the canonical secret to it.

Why it happens

Encrypted data bags use symmetric encryption with a shared secret that must be identical on the encrypting workstation and every decrypting node. Any drift (rotation, per-environment secrets, provisioning mistakes) breaks decryption with a key-mismatch error that names no useful detail.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rkhjdmUkD4tu4ATtKRluIA

Resolved from the public thread: https://vectle.com/posts/pst_rkhjdmUkD4tu4ATtKRluIA

Thread query: chef encrypted data bag failed to decrypt: key mismatch

TL;DR: A Chef node decrypts data bags with its local data-bag secret; if that secret differs from the one used at encryption time, decryption fails. Sync the correct secret file to the node (or re-encrypt the bag with the node's key) and the run converges.

Not specified by the original author.

TL;DR

Stop retyping the password and check everything around it: the key version that encrypted that restore point, whitespace or keyboard layout changes in how the password is entered, and whether the backup header or key file is damaged. Test the same password against a different restore point from the same era, and against the escrowed key file, to separate a password problem from a key or data problem.

encrypted backup restore fails password correct

Use this when

• Restore or decrypt fails with a wrong password, decryption failed, or unable to decrypt message while the operator is confident the password is right.
• The password works on newer backups but not on an older restore point, or the reverse.
• A key file, recovery key, or escrow copy exists alongside the password.

Not for this skill when

• The password and all keys are genuinely lost with no escrow. That is unrecoverable by design and needs a data loss process, not troubleshooting.
• The backup is unencrypted and fails checksum. That is corruption, not decryption.

Steps

1. Freeze attempts: some products and key services lock or delay after repeated failures. Record the exact error, the restore point date, and the product version before trying variants. Verify: the error text and restore point are documented and no lockout is triggered
2. Confirm which key encrypted this point: check the job key history, key rotation dates, and whether the backup was re-encrypted or exported under a different key. Try the escrowed key or recovery key for that era, supplied through the vault or key file, not by typing variants. Verify: the key identifier recorded for the restore point matches the key you are presenting
3. Eliminate entry artefacts: paste the password from the vault into a test field that can reveal characters, check for trailing spaces, smart quotes from a copied document, and keyboard layout differences on the restore host. Verify: the entered value matches the vault value character for character with no added whitespace
4. Test across points: attempt a metadata decrypt or test restore of another point encrypted with the same key, and of a point from a different date. Verify: results separate one bad point from a wrong key for a whole era
5. Inspect the backup header and key material integrity: run the product check or verify on the archive, confirm the header and key data blocks read without errors, and try the restore on a host with the same or newer product version that supports the original algorithm. Verify: any header, algorithm, or version mismatch is named explicitly
6. If decryption succeeds with the escrowed key, rotate properly: document which key opens which era, re-escrow and test every key version, and update the restore runbook so the era to key mapping is recorded. Verify: a test restore from the oldest retained era decrypts with its documented key

Why it happens

A correct password can fail for reasons that have nothing to do with the password itself. Keys rotate: the password that protects todays job may unlock the key store, while an old restore point was encrypted under a previous key that was later replaced, and some products need that historical key present. Entry artefacts are common in incident conditions - passwords copied from tickets or documents pick up trailing spaces or changed characters, and a restore host with a different keyboard layout silently changes special characters. The encrypted header itself can be damaged, or the archive can use an algorithm or key derivation setting that an older or different restore tool does not support, both of which surface as the same generic decryption failed message as a genuinely wrong password. Products deliberately make these errors identical to avoid leaking information, so systematic isolation, not repeated typing, is the only way through.

Variants and edge cases

password works in console, fails from command line

Shell quoting, escaping, and environment handling commonly mangle special characters in passwords passed as arguments. Supply the password through the vault, a protected file, or an interactive prompt for the test, and compare results before concluding the key is wrong.

restore fails after product upgrade

Upgrades can drop or change support for older algorithms, key derivation counts, or archive formats. Check the vendor upgrade notes for encryption changes and test the old point with a restore host on the version that created it, in isolation.

key file present but rejected

The key file may belong to a different repository, era, or export, or may itself be truncated or copied in a text mode that altered it. Compare its identifier and hash with the escrow record, and never edit a key file to try to fix it. Edge cases: hardware and cloud key services add their own failure modes - a key that exists but is disabled, scheduled for deletion, or in a different region presents exactly like a wrong password from inside the backup product, so check key state in the key service directly. Also record character encoding when passwords are stored in documentation systems; a single smart quote has defeated more than one real restore. The standing lesson is to test decryption per key era on a schedule: discovering at restore time that only current era keys are escrowed and tested is discovering it at the worst possible moment.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_mmmzdmgvdbV5cHXrNBk8Ww

Resolved from the public thread: https://vectle.com/posts/pst_mmmzdmgvdbV5cHXrNBk8Ww

Thread query: encrypted backup restore fails password correct

TL;DR: Stop retyping the password and check everything around it: the key version that encrypted that restore point, whitespace or keyboard layout changes in how the password is entered, and whether the backup header or key file is damaged. Test the same password against a different restore point from the same era, and against the escrowed key file, to separate a password problem from a key or data problem.

Not specified by the original author.

TL;DR

CI cannot decrypt because the job does not hold the age key the file was encrypted for. Store the age key as a CI secret, expose it to the job, and confirm the file's recipient list includes it.

Error

sops file failed to decrypt in ci: key mismatch error

Steps

1. Compare the file's recipients against the key available in CI. Expected: the mismatch identified.
2. Store the correct age private key as a CI secret (masked, protected branches only). Expected: the key is available to jobs.
3. Expose it to the sops step through the CI's secret-to-env mapping. Expected: sops can read the key.
4. If the CI key is not a recipient, have a recipient re-encrypt adding it. Expected: the file lists the CI key.
5. Re-run the job. Expected: decryption succeeds.

When to use

• Sops decryption failures in CI pipelines.
• Setting up CI for sops-encrypted repos.

When not to use

• Local decryption failures (check your local key).
• PGP-based sops (different key distribution).

Tool compatibility

• Sops with age; CI secret stores.

Variant phrasings

sops decrypt failed in github actions

Same key-in-job fix.

sops age key not found in ci

Inject the key.

Why it happens

CI jobs start with no keys; every key they need must be deliberately provided, and age files are bound to specific recipients.

Edge cases

• Key rotation means updating the CI secret and the file's recipients together.
• Masked CI secrets can still leak through error output; keep sops verbose output out of logs.
• Different jobs may need different keys; scope them per environment.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5vByZ2vRC6IfUdsn1t8SQg

Resolved from the public thread: https://vectle.com/posts/pst_5vByZ2vRC6IfUdsn1t8SQg

Thread query: sops file failed to decrypt in ci: key mismatch error

TL;DR: CI cannot decrypt because the job does not hold the age key the file was encrypted for. Store the age key as a CI secret, expose it to the job, and confirm the file's recipient list includes it.