sendgrid 401 'permission denied, wrong credentials' - agent's API key was rotated and nobody updated it

TL;DR

Treat auth as a lease that can die at any hour, not a one-time setup. On 401/invalid_grant, refresh the token once, replay the failed request, and if it fails again, stop and page a human. Retry loops on dead credentials burn quota and get accounts locked.

The query

sendgrid 401 'permission denied, wrong credentials'  -  agent's API key was rotated and nobody updated it

Use this when

• 401/403/invalid_grant errors, expired sessions, revoked OAuth grants, rotated keys
• an agent looping on login or retrying failed sends for hours

Not for

• rate-limit 429s (those are quota, not auth)
• provider-side outages returning 5xx

Steps

Step 1: Confirm it is auth, not a limit or outage

One 401 with 'invalid_token' or 'expired' is auth. Repeated 401s after a refresh, or a 403 mentioning scopes, means the credential or its permissions changed. Check the provider's status page and the app's permission page before touching code.

**Expected: you know whether the token died, the scopes shrank, or the credential was revoked.

Step 2: Refresh once, then replay the failed request

Use the stored refresh token to mint a new access token, persist it, and replay exactly the request that failed. One refresh attempt only. If the refresh itself fails (invalid_grant), the grant is dead and no amount of retrying will revive it.

**Expected: one token refresh; the replayed request returns 200, or you move to step 3.

Step 3: Stop the retry loop and park the work

If the refresh failed, halt the job and move pending work to a holding queue. An agent that keeps clicking login or retrying sends with a dead credential burns daily quota and can trigger suspicious-activity locks.

**Expected: zero further auth attempts; pending work parked with a clear 'auth dead' reason.

Step 4: Re-auth through the proper channel

Rotate the credential the way the provider intends: re-run the OAuth flow, generate a new app password or API key, and store it in the secret store, not in code or env files. Verify the new credential with a single read-only call before resuming writes.

**Expected: one read-only probe succeeds with the new credential before any send resumes.

Step 5: Add the lease-expiry guard

Track token expiry and refresh proactively at ~80 percent of TTL. Alert when a refresh fails once, and cap auth retries at 2 so a dead credential can never loop for hours.

**Expected: tokens refresh before expiry; a failed refresh pages someone within minutes.

Why it happens

Auth dies on schedules the agent does not know about: 2-hour token TTLs, rotated keys, revoked grants, reduced scopes. Agents built on 'auth worked at setup' keep running with a dead lease and turn one expired token into thousands of failed requests.

Edge cases

• A refresh token can be revoked by re-authenticating elsewhere; handle invalid_grant as terminal.
• Scope reductions look like auth failures but need an admin to re-grant, not a new token.
• 2FA-gated sessions cannot be refreshed headlessly; plan for a human in that loop.

Finding this skill

This skill was written for sales-outreach (SDR) agents hitting the failure above. Search Vectle for the exact error text or symptom to find it and related skills.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qfrw6CkB6kqVMXWCofG0qA