## TL;DR
InvalidClientTokenId means the access key ID does not exist: a typo, a deleted key, or the wrong credentials file/profile being picked up. In automation the usual story is environment variable precedence: stale AWS_ACCESS_KEY_ID in the environment shadowing the intended credentials, or the wrong profile selected. Print which key ID is being used (it is not secret) and trace where it came from.

## The query
```text
AWS "InvalidClientTokenId" in automation: credential chain debugging
```

## Use this when
- Automation fails with InvalidClientTokenId
- Credentials work locally but fail in CI
- Debugging credential precedence
- After credential rotation

## Not for when
- IAM "not authorized" errors (key valid, permissions lacking)
- Expired session tokens (different error)
- STS assume-role failures

## Steps

### Step 1: Identify which key ID is being used
Log or print the access key ID the failing call uses (key IDs are not secret). Compare it against the expected key. A completely unexpected ID means the wrong credential source is winning.
Expected output: the actual key ID in use, compared to the intended one.

### Step 2: Walk the credential provider chain
Check each source in precedence order: environment variables, shared credentials file profiles, container/IAM roles. The winner is the first source with values; stale values in a high-precedence source shadow everything below.
Expected output: the winning credential source identified.

### Step 3: Check for stale environment variables
Look for AWS_ACCESS_KEY_ID set in the CI environment, container env, or shell profile from an old rotation. Stale env vars are the number one cause in automation; they override fresher file-based credentials silently.
Expected output: stale variables found and removed, or ruled out.

### Step 4: Verify the key exists and is active
Check in IAM that the key ID exists and is active. Deleted or deactivated keys produce exactly this error; rotation that deleted the old key before automation picked up the new one is the classic sequence.
Expected output: the key confirmed existing and active, or the deletion found.

### Step 5: Prefer IAM roles over static keys
Where possible, eliminate static keys: use IAM roles for EC2, ECS task roles, or OIDC federation for CI. Roles cannot suffer InvalidClientTokenId from stale keys because there are no keys.
Expected output: the credential source moved to roles, removing the failure class.

## Provenance

Resolved from the public thread: https://vectle.com/posts/pst_leRZ4OCY0RUgnwcb6mY_Bg