agent's webhook signature verification failed: clock skew between agent and sender

agent's webhook signature verification failed: clock skew between agent and sender

TL;DR

Most webhook signature schemes include a timestamp and reject anything older than a few minutes to stop replay attacks. If the agent's clock has drifted, or the tolerance is set too tight, legitimate webhooks fail verification. First prove the failure is the timestamp and not the signature by verifying the HMAC alone, then widen the tolerance to 5 minutes and sync the clock with NTP.

agent's webhook signature verification failed: clock skew between agent and sender

Use this when

• Webhook verification fails intermittently, worse after the receiver host has been up a long time
• The same webhook verifies on one host but not another
• Signature checks pass in the sender's test tool but fail on your receiver
• You see timestamp-too-old or tolerance-exceeded style errors

Not for this skill when

• The signature itself never verifies (that is a wrong secret or a payload-mangling problem, usually a proxy rewriting the body)
• You are designing a new webhook signature scheme from scratch
• The sender does not sign webhooks at all

Steps

1. Split the failure: verify the HMAC signature over the raw request body while ignoring the timestamp. Read the signing secret from your secret store, never hardcode it. Expected: the signature check passes on its own, which proves the timestamp check is the part failing.

2. Measure the skew: log the sender's timestamp header next to your clock at receipt time.

   import time
   sent = int(sender_timestamp_header)  # unix seconds from the webhook header
   skew = abs(time.time() - sent)
   print("skew seconds:", skew)

   Expected: you see a concrete number, for example 214 seconds.

3. Set the timestamp tolerance to 300 seconds (5 minutes), the common default across providers. Reject anything older, and also reject timestamps far in the future. Expected: webhooks inside the window verify; old replays still fail.

4. Sync the host clock with NTP (chrony or ntpd) and alert if the offset ever exceeds your tolerance again. Containers inherit the host clock, so fix the host. Expected: the reported clock offset is a few milliseconds.

5. Replay one of the failed webhooks and confirm it verifies end to end. Expected: 200 response, signature valid, timestamp within window.

Variant: Stripe webhook signature timestamp too old

Stripe's tolerance is 300 seconds by default. Same fix: confirm the failure is the timestamp check, sync the clock, keep the default tolerance. Do not disable timestamp checking to make it pass.

Variant: Slack request timestamp verification failed

Slack signs with its signing secret and includes a request timestamp header. Verify the timestamp is within 5 minutes before comparing signatures, and use a constant-time compare for the signature itself.

Variant: webhook InvalidSignatureException clock skew

AWS-style errors with this name are almost always the timestamp. Measure the skew first; if it is near zero, the problem is the secret or the signed payload construction instead.

Why this happens

Timestamped signatures stop replay attacks: a captured webhook cannot be reused hours later. That protection assumes both clocks agree within the tolerance. VM and container clocks drift without NTP, so the window silently closes on legitimate traffic.

Edge cases and pitfalls

• Widening tolerance past 5 minutes re-opens the replay window; fix the clock instead of widening further.
• Some senders use millisecond timestamps; converting wrong turns a fresh webhook into one from 1970. Check the unit.
• Proxies or middleware that parse and re-serialize the JSON body break the signature even when the timestamp is fine; always verify against the raw body bytes.
• Unix time has no DST, but a host with the wrong timezone plus naive datetime math can still produce skew; compare unix timestamps, not local times.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_eg6H-Jg60fXd83og6b187A