agent missed user-reported phish alerts when graph webhook notifications stopped after certificate rotation

TL;DR

Restore the notification path and prove it with a live event: check the webhook endpoint certificate chain after rotation, confirm Graph subscriptions still exist and point at the right address, recreate any subscription that failed validation, and add an expiry and heartbeat monitor, because Graph stops delivering to an endpoint it cannot validate and says very little about it. A silent agent after a certificate change is a delivery failure until proven otherwise.

agent missed user-reported phish alerts when graph webhook notifications stopped after certificate rotation

Use this when

• User reported phish alerts stopped reaching the triage agent after a certificate rotation on its endpoint.
• The agent itself is running and its polling or health checks look normal.
• Graph subscription validation or delivery errors appear around the rotation time.

Not for this skill when

• Reports stopped because users stopped reporting. Check the submissions portal volume before the agent pipeline.
• The agent queue is backed up and dropping alerts. That is a consumer capacity problem, not notification delivery.

Steps

1. List the current Graph subscriptions for the reporting mailbox and note their expiry times and notification addresses. Work from a copied message or a test send first so production mail stays untouched while you compare the before and after state. Record the timestamp, the sending service, and the exact header or log line that changed, because deliverability and phishing faults often look identical until that evidence is lined up side by side in one place. Verify: each subscription is listed with expiry and address, or its absence is confirmed
2. Validate the endpoint from outside: fetch its certificate chain and confirm the full chain is served and matches the host name Graph calls. Do not change more than one setting at a time here. If you fix the record, the connector, and the policy in the same window you will not know which change cleared the fault, and the next regression will take just as long to isolate. Save the working configuration once this step checks out before moving on. Verify: an external check shows a complete, matching certificate chain
3. Recreate a test subscription and watch the validation request arrive and succeed in the endpoint logs. Use the receiving side view for this check rather than the sender console alone. Sender dashboards often report a local success while the recipient gateway applied a different verdict, rewrote a header, or dropped the message at a later hop. The recipient header, portal log, or message trace entry is the evidence that counts. Verify: the validation event is logged and the subscription becomes active
4. Send a controlled test report and trace it from submission to the agent queue. Keep any exception you create narrow and time boxed. Scope it to the one sender, domain, or message type you are fixing, note the owner and the review date in the change record, and remove it once the root fix is in place so a temporary workaround does not quietly become a permanent hole in the filter. Verify: the test alert reaches the queue with a normal timestamp
5. Add monitoring for subscription expiry and a synthetic report heartbeat, alerting when no notification arrives in the expected window. Retest with a fresh message through the exact same path as the failing one, not a shortcut path. A new test from a different client, gateway, or mailbox can pass while the original route is still broken, which is the most common reason these faults get closed early and then reopen within a day. Verify: the monitor exists and a test alert from it is received

Why it happens

Change notifications depend on the endpoint passing Graph validation over TLS every time a subscription is created or renewed. A certificate rotation that drops an intermediate, mismatches a name, or happens on only one of several endpoint replicas makes validation fail intermittently or fully, and subscriptions then lapse at expiry without drama. The agent code is usually innocent; the contract between the platform and the endpoint broke at the transport layer during routine maintenance.

Variants and edge cases

agent missed phish alerts graph webhook notifications stopped certificate rotation

The rotation linked silence in full.

graph change notifications not delivered after cert change subscription validation failed

The validation failure variant.

webhook subscription expired silently triage agent no alerts monitoring

The monitoring gap variant for the follow up fix.

Compatibility: Microsoft Graph change notifications and subscription lifecycle, TLS certificate management for webhook endpoints.

Edge cases to check:

• Rotate certificates on webhook endpoints during a watched window and immediately run a synthetic event; it is the cheapest proof the path survived.
• Track subscription expiry like certificate expiry, in the same calendar, because both fail quietly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pr3B36GTtJpNrKoXqraeEw