Lever webhook "403 forbidden" - the agent's signature check rejects valid events after a secret rotation
Fixes Lever webhook 403s from stale signature secrets after rotation. Use when valid webhooks get rejected post-rotation. Key trigger: 403s on webhooks right after a secret change.
Lever webhook "403 forbidden" - the agent's signature check rejects valid events after a secret rotation
TL;DR
The webhook secret was rotated, but the agent's signature check still uses the old one, so valid events get rejected as forgeries. Update the verification secret atomically with the rotation, and support a brief overlap window where both secrets verify. Rejected valid events are lost pipeline updates.
Lever webhook 403 Forbidden (signature check failing after secret rotation)Steps
Confirm the cause: rotation timestamp vs first 403. Expected: They coincide.
Update the agent's verification secret to the new value. Expected: Valid events verify again.
Add an overlap window: accept signatures from the old or new secret during rotation. Expected: Rotations stop dropping events.
Reconcile the rejected window: re-fetch the events missed while verification failed. Expected: The pipeline is whole.
Make secret rotation a two-phase process (add new, remove old) with the overlap built in. Expected: Future rotations are lossless.
When to use
- webhooks 403 after a secret rotation
- the endpoint verifies signatures
- rotations are manual
When not to use
- the 403 predates the rotation (different cause)
- the events are genuinely forged (then the 403 is correct)
- the endpoint does not verify signatures (check the auth mechanism)
Tool compatibility
- Lever webhooks
- HMAC signature verification
Variant phrasings
- "lever webhook 403 forbidden"
- "webhook signature rotation 403"
- "lever webhook secret rotated"
Why it happens
Signature verification compares the event's HMAC against your stored secret. Rotate the secret on one side and every legitimate event looks forged to the other. The overlap window exists because rotation is never truly atomic across two systems.
Edge cases
- the overlap window must be short; two valid secrets doubles the forgery surface
- missed webhooks need reconciliation; the events do not wait
- log verification failures with the key ID when the provider sends one
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_VNgpGTUxY-TKVe-mTub8Ew