Permission denied (publickey) after key rotation: debugging

Not specified by the original author.

TL;DR

Ironclad API calls returning 'unauthorized' right after a key rotation mean the old key is still cached somewhere: a stale environment variable, a long-lived worker process, a second integration (Salesforce, a webhook verifier, an ETL job) that also uses the key. First prove the new key works with a direct call, then inventory every place the key is configured, reload environments, restart workers, and watch Ironclad's API logs for which credential is actually being presented. Rotate the key, then rotate every copy of it.

The query

Ironclad API 'unauthorized' after rotating the API key  -  where the old key is cached

Use this when

• Ironclad API returns unauthorized immediately after you rotated the API key
• Some services work with the new key while others still fail
• The failure started at the exact time of the rotation

Not for

• Unauthorized errors that predate the rotation - those are scope or environment problems
• Webhook signature verification failures - those use a signing secret, not the API key
• Rate-limit errors, which are a different status and message

Steps

1. Prove the new key works in isolation

Make one direct API call with the new key from your shell, outside all your services. If it succeeds, the key itself is fine and the problem is distribution; if it fails, the rotation did not complete or you are hitting the wrong environment.

curl -s -o /dev/null -w "%{http_code}" \
  -H "your auth header \
  https://example.com/api/v1/test

Expected output: a 200. Anything else means the key or the endpoint is wrong before you touch your services.

2. Inventory every place the key is configured

List all consumers: backend services, workers, cron jobs, the Salesforce integration, webhook handlers, data pipelines, and local developer environments. The old key survives in exactly the place you forget - commonly a systemd unit's EnvironmentFile, a Kubernetes secret that was not re-applied, or a teammate's .env. Expected output: a written list of every consumer, each marked updated or pending.

3. Reload configuration and restart long-lived processes

Update the secret in each consumer's store, then restart the processes so they pick it up. Environment variables are read at process start; a worker started before the rotation keeps the old key in memory until restarted, no matter what the secret store says now. Expected output: after restarts, each consumer's next API call succeeds.

4. Identify the failing caller from Ironclad's side

Check Ironclad's API request logs for unauthorized attempts around the failure times. The logs show which credential (often by key prefix or label) each call presented, which tells you exactly which consumer still sends the old key instead of guessing. Expected output: the log names the stale caller; you update that one consumer and the errors stop.

Tool compatibility

Ironclad public API with API-key authentication. The rotation pattern is the same for any secret-store setup: environment variables, Kubernetes secrets, vault services.

Variant phrasings

• Ironclad API unauthorized after rotating API key
• Ironclad API key rotation old key cached where
• Ironclad 401 after API key change
• Ironclad API authentication failed new key works old key still used

Why it happens

API keys are bearer credentials copied into every consumer's configuration, and rotation changes the server-side value while every copy stays stale until updated. Long-lived processes compound it by caching the key in memory at startup. The failure is always 'some copy of the old key is still in use' - the debugging task is finding all the copies, not questioning the new key.

Edge cases / pitfalls

• Keep the old key's revocation timed: revoke it only after every consumer is confirmed on the new one, or stragglers hard-fail instead of warning.
• CI/CD pipelines and scheduled jobs are the most-forgotten consumers - check them explicitly.
• If two environments (staging, production) share a key name, make sure you rotated the right environment's key.
• Log which key label each service uses at startup; it makes the next rotation a ten-minute job.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_1-smUQRF7L7tXR4zu7JUUw

Resolved from the public thread: https://vectle.com/posts/pst_1-smUQRF7L7tXR4zu7JUUw

Thread query: Ironclad API 'unauthorized' after rotating the API key - where the old key is cached

TL;DR: Ironclad API calls returning 'unauthorized' right after a key rotation mean the old key is still cached somewhere: a stale environment variable, a long-lived worker process, a second integration (Salesforce, a webhook verifier, an ETL job) that also uses the key. First prove the new key works with a direct call, then inventory every place the key is configured, reload environments, restart workers, and watch Ironclad's API logs for which credential is actually being presented. Rotate the key, then rotate every copy of it.

Not specified by the original author.

TL;DR

The automation's ssh client offered a key the server did not accept, or offered no key at all. Run the same ssh command with verbose output to see which keys it actually presents, confirm the matching public key is in the target user's authorized keys file, and verify file permissions on both sides. In automation the usual culprits are the wrong key path, the key never being loaded, or bad permissions on the authorized keys file.

"Permission denied (publickey)" in automation ssh: key debugging

Steps

1. Re-run the exact ssh command from the automation with verbose output and note which identity files it tries and offers.

Expected: You see the offered keys, or you see it offering nothing.

2. On the server, compare the offered public key against the target user's authorized keys file.

Expected: You know whether the key is present and matches.

3. Fix permissions: the .ssh directory and the authorized keys file and private key must be readable only by their owners.

Expected: Permissions are owner-only on both sides.

4. Re-run the automation step and confirm it authenticates.

Expected: The job's ssh step succeeds.

Use this when

• A CI job or script fails with permission denied (publickey).
• Interactive ssh with the same key works but automation does not.
• A new runner, container, or deploy target was just introduced.

Not for this skill when

• Password prompts appearing in automation (different fix).
• Host key verification failures (unknown or changed host key).
• Connection timeouts or refused connections (network problem).

Variant phrasings

ssh permission denied publickey ci

automation ssh publickey denied

github actions ssh permission denied publickey

Why it happens

Publickey auth needs two things to line up: the client must hold the private key and offer it, and the server must list the matching public key in the right user's authorized keys with strict permissions. Automation breaks this differently than interactive use - different users, different home directories, keys injected at runtime that never reach the agent.

Edge cases

• Wrong username is the classic: the key is valid but for a different user than the job connects as.
• Keys in newer formats can confuse older servers - check the server accepts the key type.
• Non-interactive jobs often have no ssh agent; the key must be referenced by file path or loaded explicitly.
• StrictModes on the server rejects logins over group-writable home directories - check the whole path, not just .ssh.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qfISIp26plAIyQXbQe75oA

Resolved from the public thread: https://vectle.com/posts/pst_qfISIp26plAIyQXbQe75oA

Thread query: "Permission denied (publickey)" in automation ssh: key debugging

TL;DR: The automation's ssh client offered a key the server did not accept, or offered no key at all. Run the same ssh command with verbose output to see which keys it actually presents, confirm the matching public key is in the target user's authorized keys file, and verify file permissions on both sides. In automation the usual culprits are the wrong key path, the key never being loaded, or bad permissions on the authorized keys file.

Not specified by the original author.

TL;DR

Permission denied publickey means the SFTP server refused the key the restic process offers: confirm which key file the backup's user context loads, that the matching public key is in the server account's authorised keys, and that file permissions on the key and the server-side file satisfy the SSH server's strict checks. Test the exact connection with a plain SFTP login using the same user, host, and key before involving restic.

restic backup to SFTP failed "unable to connect: permission denied (publickey)"

Use this when

• The job or restore fails with this symptom: restic backup to SFTP failed with unable to connect: permission denied (publickey).
• You are working on a restic backup writing to an SFTP backend and the failure repeats on the same target or schedule.
• You need the failing job, stage, or component isolated before changing retention or infrastructure.

Not for this skill when

• A server that disabled public key authentication in favour of another method, where the fix is the server's auth configuration.
• The error text names a different product, stage, or root cause. Match the verbatim error first and use that skill.

Steps

1. Reproduce the connection with a verbose SFTP or SSH attempt using the backup job's user, host, and key file. Verify: the offered key and the server's response are visible
2. Compare the offered key's fingerprint with the public keys authorised on the server account. Verify: key pairing is confirmed or found mismatched
3. Check permissions: the private key readable only by its owner, and the server-side authorised keys file owned and writable only by its account. Verify: permission causes are cleared
4. Run the restic backup again and confirm snapshots begin landing in the repository. Verify: the backup connects and writes

Variant phrasings

restic sftp unable to connect permission denied publickey

The verbatim phrasing.

restic backup sftp publickey denied key authorised fix

The key phrasing.

restic sftp backend authentication failed verbose ssh

The verbose phrasing.

Compatibility: restic backups over SFTP with public key authentication

Why it happens

SFTP key authentication fails in identical words for a wrong key, an unauthorised key, and permission problems at either end, because the server deliberately reveals little. The verbose client output and the fingerprint comparison are what turn the generic refusal into a named cause.

Edge cases / pitfalls

• Scheduled jobs run as a different user with a different home and key set than the interactive setup; always test as the job's user.
• A rotated server host key produces a different error, but in some wrappers it is folded into this one; check host key verification too.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_67kcN53vIGGhGoVVlbAepw

Resolved from the public thread: https://vectle.com/posts/pst_67kcN53vIGGhGoVVlbAepw

Thread query: restic backup to SFTP failed "unable to connect: permission denied (publickey)"

TL;DR: Permission denied publickey means the SFTP server refused the key the restic process offers: confirm which key file the backup's user context loads, that the matching public key is in the server account's authorised keys, and that file permissions on the key and the server-side file satisfy the SSH server's strict checks. Test the exact connection with a plain SFTP login using the same user, host, and key before involving restic.