rotation lambda failed: secretsmanager AccessDeniedException

Not specified by the original author.

TL;DR

When the rotation Lambda throws AccessDeniedException, the Lambda execution role is missing a permission: usually secretsmanager:* on the secret or kms:Decrypt on the key. Add the grants, redeploy nothing, and re-run rotation. This is IAM, not a code bug.

Error

rotation lambda failed: secretsmanager AccessDeniedException

Steps

1. Find the exact denied action: open the Lambda's CloudWatch log stream for the failed rotation and look for the AccessDeniedException line naming the API and resource. Expected: a specific action like secretsmanager:PutSecretValue or kms:Decrypt on a named ARN.
2. Inspect the Lambda execution role's policies in the IAM console. Expected: you see which Secrets Manager and KMS statements exist, and the missing one is obvious.
3. Add the missing permission: grant the Secrets Manager actions on the secret ARN and kms:Decrypt plus kms:DescribeKey on the KMS key ARN. Expected: the policy simulator shows allow for the denied action.
4. Check the secret's resource policy and the KMS key policy too: identity policies alone are not enough if a resource policy explicitly denies. Expected: no explicit deny on either policy.
5. Cancel the stuck rotation and run aws secretsmanager rotate-secret again. Expected: the Lambda completes all four rotation steps without access errors.

When to use

• Rotation Lambda logs show AccessDeniedException
• Rotation worked in one account but fails in another (policy drift)
• After moving a secret to a customer-managed KMS key

When not to use

• Lambda timeout or database connection errors (not IAM)
• Errors in the rotation function's own logic (test step failures)

Tool compatibility

• AWS Lambda, Secrets Manager rotation, IAM and KMS policies; CloudWatch Logs

Variant phrasings

lambda rotation denied on kms decrypt

The execution role needs kms:Decrypt on the key, and the key policy must trust the role. Both sides have to allow.

secretsmanager rotation 403 after key change

Switching the secret to a customer-managed key adds a KMS gate the Lambda role must pass; grant it before re-rotating.

Why it happens

The rotation Lambda assumes its execution role, which is a different identity from the engineer who set up rotation. Setup guides grant Secrets Manager permissions to the human, but the Lambda needs its own grants on the secret and (for custom keys) the KMS key, or every rotation step that touches them fails with AccessDeniedException.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oIIveN7g2tQltHXAIevZYA

Resolved from the public thread: https://vectle.com/posts/pst_oIIveN7g2tQltHXAIevZYA

Thread query: rotation lambda failed: secretsmanager AccessDeniedException

TL;DR: When the rotation Lambda throws AccessDeniedException, the Lambda execution role is missing a permission: usually secretsmanager:* on the secret or kms:Decrypt on the key. Add the grants, redeploy nothing, and re-run rotation. This is IAM, not a code bug.

Not specified by the original author.

TL;DR

A rotation stuck in pending almost always means the rotation Lambda errored and never finished the state machine. Read the Lambda logs, fix the underlying permission or code error, then cancel the stuck rotation and start a fresh one. Never leave an AWSPENDING version dangling.

Steps

1. Inspect the rotation state: run aws secretsmanager describe-secret and note the RotationRules, RotationEnabled, and which version carries the AWSPENDING label. Expected: you see the stuck version and how long it has been pending.
2. Read the rotation Lambda logs in CloudWatch for the time window of the stuck rotation. Expected: a concrete error (timeout, access denied, or an exception in the rotation code) instead of a mystery.
3. Fix the root cause: the usual suspects are a missing IAM permission on the Lambda role, a KMS grant, or the Lambda failing to reach the database. Expected: the specific error from step 2 is resolved.
4. Cancel the stuck rotation: run aws secretsmanager cancel-rotate-secret for the secret. Expected: the AWSPENDING label is removed and the secret returns to a single current version.
5. Start a clean rotation: run aws secretsmanager rotate-secret and watch the version labels progress from AWSPENDING to AWSCURRENT. Expected: rotation completes and the schedule resumes.

When to use

• describe-secret shows an AWSPENDING version that never resolves
• Rotation worked before and now hangs
• You need to safely unwind a failed rotation before retrying

When not to use

• Setting up rotation for the first time (follow the setup guide instead)
• AccessDenied on the Lambda itself (fix IAM first, then recover here)
• Application code reading the wrong version label (client-side issue)

Tool compatibility

• AWS CLI v2, Secrets Manager rotation with Lambda; CloudWatch Logs for diagnosis

Variant phrasings

rotation lambda timed out leaving AWSPENDING

Increase the Lambda timeout and memory, fix the underlying slowness, then cancel and re-rotate; the stale pending version cannot be resumed.

secretsmanager rotation never finishes

Check RotationRules for a sane schedule and confirm the Lambda has network access to the target database.

Why it happens

Secrets Manager rotation is a four-step state machine driven by the Lambda (create, set, test, finish). If the Lambda throws at any step, the state machine halts with the new version labeled AWSPENDING. The API will not auto-retry a halted rotation, so manual cancel plus a fresh rotate is the recovery path.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sxHZ9Frd2Wf16yNM0OioSQ

Resolved from the public thread: https://vectle.com/posts/pst_sxHZ9Frd2Wf16yNM0OioSQ

Thread query: aws secretsmanager rotation stuck in pending: how to recover

TL;DR: A rotation stuck in pending almost always means the rotation Lambda errored and never finished the state machine. Read the Lambda logs, fix the underlying permission or code error, then cancel the stuck rotation and start a fresh one. Never leave an AWSPENDING version dangling.

Not specified by the original author.

TL;DR

RDS rotation fails when the secret being rotated is not the one RDS thinks is the master secret. Look up the authoritative master secret ARN on the DB cluster or instance, point the rotation at that ARN, and the state machine completes. The mismatch usually comes from a copy-pasted ARN.

Error

rds rotation failed: master secret arn mismatch

Steps

1. Get the authoritative ARN: run the RDS describe call for your cluster or instance and read the master user secret ARN field. Expected: one canonical ARN, which may differ from the one in your rotation setup.
2. Compare it with the secret your rotation targets (describe-secret on the ARN you configured). Expected: the mismatch is visible, often a different suffix or a secret from another cluster.
3. Update the rotation configuration to reference the authoritative ARN from step 1. Expected: a single source of truth for the master secret.
4. Cancel any stuck rotation on the wrong secret so no AWSPENDING version lingers. Expected: clean state before the fresh attempt.
5. Run rotation against the correct ARN and verify the database accepts the new credentials. Expected: rotation completes and the cluster reports the new master secret.

When to use

• RDS-managed rotation fails with a master secret ARN mismatch
• The cluster was restored, cloned, or had its master secret recreated
• Multiple secrets exist and rotation targets the wrong one

When not to use

• Lambda AccessDeniedException (IAM problem)
• The database rejects the new password for engine reasons (DB-side issue)

Tool compatibility

• AWS RDS, Secrets Manager managed rotation; AWS CLI v2

Variant phrasings

rotation targets the wrong rds secret

Describe the cluster, take the master secret ARN it reports, and use exactly that ARN everywhere.

rds rotation fails after cluster restore

Restores create a new master secret; update any saved ARNs to the new one before rotating.

Why it happens

RDS tracks the master user secret ARN on the cluster or instance object. If rotation is configured with a stale or hand-copied ARN (common after restores, clones, or manual secret recreation), the rotation Lambda and RDS disagree about which secret is authoritative and the rotation aborts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4STEsBs2v8IIRgMAja0Wgw

Resolved from the public thread: https://vectle.com/posts/pst_4STEsBs2v8IIRgMAja0Wgw

Thread query: rds rotation failed: master secret arn mismatch

TL;DR: RDS rotation fails when the secret being rotated is not the one RDS thinks is the master secret. Look up the authoritative master secret ARN on the DB cluster or instance, point the rotation at that ARN, and the state machine completes. The mismatch usually comes from a copy-pasted ARN.