AWS "InvalidClientTokenId" in automation: credential chain debugging

TL;DR

InvalidClientTokenId means the access key ID does not exist: a typo, a deleted key, or the wrong credentials file/profile being picked up. In automation the usual story is environment variable precedence: stale AWSACCESSKEY_ID in the environment shadowing the intended credentials, or the wrong profile selected. Print which key ID is being used (it is not secret) and trace where it came from.

The query

AWS "InvalidClientTokenId" in automation: credential chain debugging

Use this when

• Automation fails with InvalidClientTokenId
• Credentials work locally but fail in CI
• Debugging credential precedence
• After credential rotation

Not for when

• IAM "not authorized" errors (key valid, permissions lacking)
• Expired session tokens (different error)
• STS assume-role failures

Steps

Step 1: Identify which key ID is being used

Log or print the access key ID the failing call uses (key IDs are not secret). Compare it against the expected key. A completely unexpected ID means the wrong credential source is winning. Expected output: the actual key ID in use, compared to the intended one.

Step 2: Walk the credential provider chain

Check each source in precedence order: environment variables, shared credentials file profiles, container/IAM roles. The winner is the first source with values; stale values in a high-precedence source shadow everything below. Expected output: the winning credential source identified.

Step 3: Check for stale environment variables

Look for AWSACCESSKEY_ID set in the CI environment, container env, or shell profile from an old rotation. Stale env vars are the number one cause in automation; they override fresher file-based credentials silently. Expected output: stale variables found and removed, or ruled out.

Step 4: Verify the key exists and is active

Check in IAM that the key ID exists and is active. Deleted or deactivated keys produce exactly this error; rotation that deleted the old key before automation picked up the new one is the classic sequence. Expected output: the key confirmed existing and active, or the deletion found.

Step 5: Prefer IAM roles over static keys

Where possible, eliminate static keys: use IAM roles for EC2, ECS task roles, or OIDC federation for CI. Roles cannot suffer InvalidClientTokenId from stale keys because there are no keys. Expected output: the credential source moved to roles, removing the failure class.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstleRZ4OCY0RUgnwcb6mYBg