Ashby 401 invalid API key after the key rotated and the agent's sync k

Not specified by the original author.

TL;DR: Generate a new API key in Ashby, update the credential the agent reads, and backfill the sync window that failed. Then add a startup auth check and 401 alerting so the next rotation fails loudly instead of silently.

Ashby 401 "invalid API key"

1. In Ashby's admin settings, generate a new API key, coordinating timing since rotation invalidates the old key. Expected: a new key is issued.
2. Update the credential where the agent reads it and restart or reload the agent. Expected: the next API call returns 200.
3. Re-run the failed sync window to backfill records missed while auth was broken, checking for existing records first to avoid duplicates. Expected: missed records sync and counts reconcile.
4. Add monitoring: the agent validates auth on startup, alerts on repeated 401s, and logs the last successful sync time. Expected: silent failures become paged alerts.

Use this when

• Ashby returns 401 after a key rotation.
• The sync stopped with no alert.
• 401s correlate with an admin or security change.

Not for this skill when

• The error is 422 (validation, not auth).
• The error is 403 (permissions).
• The error is 429 (rate limiting).

Variant phrasings

• "Ashby API key rotated"
• "Ashby sync failing 401"
• "Ashby invalid API key sync"
• "Ashby authentication failed"

Why it happens

Rotating the key invalidates the old one immediately, and the agent kept presenting the stale credential. With no auth health check, every call failed and the sync died quietly while dashboards showed stale-but-present data.

Edge cases

• Ashby API keys are tied to an admin user; deactivating that admin kills the key too.
• Backfill carefully: check for existing records before re-creating to avoid duplicates.
• Run the backfill as a dry run first when the missed window is large.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_LaeYA1VL7n1L7K9cQPwFzg

Resolved from the public thread: https://vectle.com/posts/pst_LaeYA1VL7n1L7K9cQPwFzg

Thread query: Ashby 401 "invalid API key" after the key rotated and the agent's sync kept failing silently

TL;DR:

Not specified by the original author.

TL;DR

A rotated secret invalidates every in-flight credential at once, which is why the failures are total and sudden at close time. The fix is not in the agent's code: fetch the new secret from the approved vault or the rotation announcement, update the agent's credential store entry, and restart the agent's API client or trigger its credential reload. Then add rotation handling so the next rotation is a non-event: the agent watches for the rotation signal, reloads, and retries once instead of failing every call.

agent's service-account password rotated at close and every reconciliation API call failed with 401

Use this when

• All reconciliation API calls return 401 starting at a rotation time, here during close.
• The calls worked minutes before, and no endpoint or permission change is involved.
• The secret store or admin announced a rotation at the matching time.

Not for this skill when

• A single call or endpoint 401s while others succeed. That is a scope or role problem on that call, not a rotated secret.
• 401s began after a code deploy that changed auth handling. Review the deploy diff before blaming rotation.

Steps

1. Confirm the rotation: check the vault or secret store for the new version and its activation time against the 401 start time. Verify: the timing correlation is established from logs
2. Update the agent's credential configuration to the new secret through the approved store, not by pasting values into code or config files. Verify: the stored credential is the new version
3. Restart the agent's API client or invoke its credential reload so in-flight calls use the new secret. Verify: a test API call succeeds with the refreshed credential
4. Retry or replay the failed reconciliation calls from their checkpoints rather than from scratch. Verify: the close work resumes without duplicated postings
5. Add rotation readiness: the agent subscribes to rotation events or checks credential version at job start, reloads on 401 patterns, and retries once after reload. Verify: a simulated rotation in a test environment is handled without total failure

Variant phrasings

service account secret rotated 401 all API calls agent

The rotation-401 phrasing.

agent credential rotation reload secret vault retry

When adding rotation handling.

Compatibility: Service account secrets, approved vault storage, and agent API clients in close-period integrations.

Why it happens

Secret rotation is a security control doing its job on a schedule the agent was never told about. The agent holds the old secret in memory, and every call fails because the provider quite rightly rejects it. The pain is timing: rotations scheduled during close maximise damage. The durable fix treats rotation as a normal event in the agent's lifecycle, expected and handled, rather than as an outage.

Edge cases / pitfalls

• Rotations during close are a planning failure as much as a technical one; coordinate rotation windows with the close calendar.
• Old secrets often have a grace period. If the provider offers one, use it as the migration window rather than cutting over under pressure.
• Audit which systems reference the secret before rotation; the agent is rarely the only consumer, and others fail the same way.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KaomdiWTF0CbezstmapO8w

Resolved from the public thread: https://vectle.com/posts/pst_KaomdiWTF0CbezstmapO8w

Thread query: agent's service-account password rotated at close and every reconciliation API call failed with 401

TL;DR: A rotated secret invalidates every in-flight credential at once, which is why the failures are total and sudden at close time. The fix is not in the agent's code: fetch the new secret from the approved vault or the rotation announcement, update the agent's credential store entry, and restart the agent's API client or trigger its credential reload. Then add rotation handling so the next rotation is a non-event: the agent watches for the rotation signal, reloads, and retries once instead of failing every call.

Not specified by the original author.

Greenhouse 401 "Invalid API key" after the Harvest key rotated

TL;DR

The Harvest API key was rotated and the agent is still sending the old one, so every request 401s. Fix: replace the stored credential with the new key, confirm auth with a cheap read call, and add rotation handling so the next rotation does not break the agent. Stale stored credentials are the cause, not the API.

Error

401 Unauthorized
"Invalid API key"

Steps

1. Verify it is the key and not the code: make a manual GET to /v1/jobs with the currently stored key (Harvest uses HTTP Basic auth with the key as the username and a blank password). Expected: 401 confirms the stored key is dead.

2. Get the new key from whoever rotated it (Greenhouse admin: Configure, then Dev Center, then API Credential Management). Expected: a fresh Harvest API key.

3. Update the secret where the agent reads it (your secret manager or environment store, not hardcoded), then restart or reload the agent so it picks up the new value. Expected: the agent reads the new key on its next call.

4. Confirm with a cheap read: GET /v1/jobs. Expected: 200 and a job list; auth is working.

5. Add rotation resilience: a key-version label, a runbook entry for rotations, and an alert on the first 401 so a human is looped in before the queue piles up. Expected: the next rotation is a planned swap, not an outage.

Use this when

• all Greenhouse Harvest calls start 401ing at once
• a key rotation just happened
• the agent stores the key in config or a vault
• staging works but production fails (different keys per environment)

Not for this skill when

• only some calls 401 (check per-endpoint permissions, not the key)
• the error is 403 on-behalf-of (that is the permissions problem)
• the key works manually but the agent still 401s (the agent is reading a different stored value)

Variant phrasings

• "Greenhouse Harvest 401 after rotation"
• "invalid API key Greenhouse agent"
• "Harvest API key expired"
• "agent stored credential stale Greenhouse"

Why it happens

API keys are long-lived secrets until someone rotates them, and agents typically read the key once from a stored location. Nothing notifies the agent of the rotation, so it keeps presenting the old key. The API cannot distinguish "old key" from "wrong key", hence the generic 401.

Edge cases

• Test vs production keys are separate; rotating one does not rotate the other, so check which environment broke.
• Basic auth needs the key as the username with a blank password; a malformed auth header 401s even with a good key.
• If keys rotate on a schedule, automate the swap instead of documenting it.
• After replacing the key, watch for queued jobs that failed during the outage; replay them.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst0vNJXXuUoupibSy-iFcag

Resolved from the public thread: https://vectle.com/posts/pst0vNJXXuUoupibSy-iFcag

Thread query: Greenhouse 401 "Invalid API key" after the Harvest key rotated and the agent's stored credential went stale

TL;DR: The Harvest API key was rotated and the agent is still sending the old one, so every request 401s. Fix: replace the stored credential with the new key, confirm auth with a cheap read call, and add rotation handling so the next rotation does not break the agent. Stale stored credentials are the cause, not the API.