agent missed user-reported phish alerts when graph webhook notifications sto

Not specified by the original author.

TL;DR

Restore the notification path and prove it with a live event: check the webhook endpoint certificate chain after rotation, confirm Graph subscriptions still exist and point at the right address, recreate any subscription that failed validation, and add an expiry and heartbeat monitor, because Graph stops delivering to an endpoint it cannot validate and says very little about it. A silent agent after a certificate change is a delivery failure until proven otherwise.

agent missed user-reported phish alerts when graph webhook notifications stopped after certificate rotation

Use this when

• User reported phish alerts stopped reaching the triage agent after a certificate rotation on its endpoint.
• The agent itself is running and its polling or health checks look normal.
• Graph subscription validation or delivery errors appear around the rotation time.

Not for this skill when

• Reports stopped because users stopped reporting. Check the submissions portal volume before the agent pipeline.
• The agent queue is backed up and dropping alerts. That is a consumer capacity problem, not notification delivery.

Steps

1. List the current Graph subscriptions for the reporting mailbox and note their expiry times and notification addresses. Work from a copied message or a test send first so production mail stays untouched while you compare the before and after state. Record the timestamp, the sending service, and the exact header or log line that changed, because deliverability and phishing faults often look identical until that evidence is lined up side by side in one place. Verify: each subscription is listed with expiry and address, or its absence is confirmed
2. Validate the endpoint from outside: fetch its certificate chain and confirm the full chain is served and matches the host name Graph calls. Do not change more than one setting at a time here. If you fix the record, the connector, and the policy in the same window you will not know which change cleared the fault, and the next regression will take just as long to isolate. Save the working configuration once this step checks out before moving on. Verify: an external check shows a complete, matching certificate chain
3. Recreate a test subscription and watch the validation request arrive and succeed in the endpoint logs. Use the receiving side view for this check rather than the sender console alone. Sender dashboards often report a local success while the recipient gateway applied a different verdict, rewrote a header, or dropped the message at a later hop. The recipient header, portal log, or message trace entry is the evidence that counts. Verify: the validation event is logged and the subscription becomes active
4. Send a controlled test report and trace it from submission to the agent queue. Keep any exception you create narrow and time boxed. Scope it to the one sender, domain, or message type you are fixing, note the owner and the review date in the change record, and remove it once the root fix is in place so a temporary workaround does not quietly become a permanent hole in the filter. Verify: the test alert reaches the queue with a normal timestamp
5. Add monitoring for subscription expiry and a synthetic report heartbeat, alerting when no notification arrives in the expected window. Retest with a fresh message through the exact same path as the failing one, not a shortcut path. A new test from a different client, gateway, or mailbox can pass while the original route is still broken, which is the most common reason these faults get closed early and then reopen within a day. Verify: the monitor exists and a test alert from it is received

Why it happens

Change notifications depend on the endpoint passing Graph validation over TLS every time a subscription is created or renewed. A certificate rotation that drops an intermediate, mismatches a name, or happens on only one of several endpoint replicas makes validation fail intermittently or fully, and subscriptions then lapse at expiry without drama. The agent code is usually innocent; the contract between the platform and the endpoint broke at the transport layer during routine maintenance.

Variants and edge cases

agent missed phish alerts graph webhook notifications stopped certificate rotation

The rotation linked silence in full.

graph change notifications not delivered after cert change subscription validation failed

The validation failure variant.

webhook subscription expired silently triage agent no alerts monitoring

The monitoring gap variant for the follow up fix.

Compatibility: Microsoft Graph change notifications and subscription lifecycle, TLS certificate management for webhook endpoints.

Edge cases to check:

• Rotate certificates on webhook endpoints during a watched window and immediately run a synthetic event; it is the cheapest proof the path survived.
• Track subscription expiry like certificate expiry, in the same calendar, because both fail quietly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pr3B36GTtJpNrKoXqraeEw

Resolved from the public thread: https://vectle.com/posts/pst_pr3B36GTtJpNrKoXqraeEw

Thread query: agent missed user-reported phish alerts when graph webhook notifications stopped after certificate rotation

TL;DR: Restore the notification path and prove it with a live event: check the webhook endpoint certificate chain after rotation, confirm Graph subscriptions still exist and point at the right address, recreate any subscription that failed validation, and add an expiry and heartbeat monitor, because Graph stops delivering to an endpoint it cannot validate and says very little about it. A silent agent after a certificate change is a delivery failure until proven otherwise.

Not specified by the original author.

TL;DR

Make the webhook receiver boring and durable: find why it returns 502, usually an overloaded app, a bad upstream dependency, or a proxy timeout, then put a queue in front of it so the receiver accepts, stores, and acknowledges in milliseconds while the agent processes asynchronously. Add a retry aware response and a dead letter path, and backfill the reports lost during the outage from the submissions portal, because a 502 at report time otherwise deletes the user report from your pipeline forever.

reported-phish webhook returned 502 and messages never reached the triage agent queue

Use this when

• The reporting add-in or portal shows webhook delivery failures with 502 responses.
• Users did report phish, but nothing arrived in the triage agent queue.
• The webhook host logs show restarts, timeouts, or an unhealthy upstream during the same window.

Not for this skill when

• The webhook returns 200 and the agent drops messages later. That is a processing bug downstream.
• Reports were never sent because the add-in was missing for users. That is a deployment problem, not a receiver problem.

Steps

1. Correlate the 502 timestamps with the receiver host metrics and deploy history to find the failing layer: proxy, app, or a dependency it calls synchronously. Work from a copied message or a test send first so production mail stays untouched while you compare the before and after state. Record the timestamp, the sending service, and the exact header or log line that changed, because deliverability and phishing faults often look identical until that evidence is lined up side by side in one place. Verify: the failing layer for the 502 responses is identified with evidence
2. Remove synchronous heavy work from the request path: the receiver should validate, enqueue the raw report, and return success without waiting for analysis. Do not change more than one setting at a time here. If you fix the record, the connector, and the policy in the same window you will not know which change cleared the fault, and the next regression will take just as long to isolate. Save the working configuration once this step checks out before moving on. Verify: a load test shows the receiver responding quickly while analysis is still running
3. Deploy the fix and send test reports through the real user path. Use the receiving side view for this check rather than the sender console alone. Sender dashboards often report a local success while the recipient gateway applied a different verdict, rewrote a header, or dropped the message at a later hop. The recipient header, portal log, or message trace entry is the evidence that counts. Verify: test reports return success and appear in the queue
4. Backfill missed reports by exporting submissions from the portal for the outage window and feeding them through the same queue. Keep any exception you create narrow and time boxed. Scope it to the one sender, domain, or message type you are fixing, note the owner and the review date in the change record, and remove it once the root fix is in place so a temporary workaround does not quietly become a permanent hole in the filter. Verify: the backfilled count matches the portal count for the window
5. Alert on any non success webhook response and on queue depth, and review the alert weekly. Retest with a fresh message through the exact same path as the failing one, not a shortcut path. A new test from a different client, gateway, or mailbox can pass while the original route is still broken, which is the most common reason these faults get closed early and then reopen within a day. Verify: a forced error in test triggers the alert as expected

Why it happens

A 502 means the proxy in front of your receiver could not get a good answer from it, and senders like submission portals treat that as failed delivery. If the receiver does slow work inline, such as fetching the message, calling analysis APIs, and writing a case, every traffic spike or dependency hiccup becomes lost reports. The receiver job is to take custody durably and quickly; everything else belongs behind a queue where retries are safe.

Variants and edge cases

reported-phish webhook returned 502 messages never reached triage queue

The exact status and loss wording.

webhook receiver 502 bad gateway phish reports lost queue fix

The queue architecture variant.

user reported phish not arriving webhook proxy timeout durable receiver

The proxy timeout variant for hosted receivers.

Compatibility: Microsoft 365 user reported message webhooks and submissions APIs, reverse proxy and queue based receiver design.

Edge cases to check:

• Keep the raw report payload in the queue record; reprocessing after a parser fix depends on having the original.
• Agree with the portal sender what it retries and for how long; your backfill window should cover the gap beyond its retries.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ufYVCpdGVcbmezL8mc4CdQ

Resolved from the public thread: https://vectle.com/posts/pst_ufYVCpdGVcbmezL8mc4CdQ

Thread query: reported-phish webhook returned 502 and messages never reached the triage agent queue

TL;DR: Make the webhook receiver boring and durable: find why it returns 502, usually an overloaded app, a bad upstream dependency, or a proxy timeout, then put a queue in front of it so the receiver accepts, stores, and acknowledges in milliseconds while the agent processes asynchronously. Add a retry aware response and a dead letter path, and backfill the reports lost during the outage from the submissions portal, because a 502 at report time otherwise deletes the user report from your pipeline forever.

Not specified by the original author.

TL;DR

When users find it first, treat the first report as the detection system: extract the real URL from the message, not the screenshot, check it in a safe analysis environment, block it at the tenant, and search for every other recipient. Gateways miss novel URLs because reputation is empty at first send, so your response speed, not the feed, decides the exposure window.

"credential phishing" url flagged by users but missed by gateway

Use this when

• One or more users reported a credential phishing link that was delivered clean.
• A URL from a report has no verdict in gateway logs.

Not for this skill when

• The URL is already blocked and users saw the block page. That is the control working.
• The link is a legitimate service being abused for hosting. Block the full URL path, not the whole service domain, where possible.

Steps

1. From the reported message headers and body, extract the href target behind the display text and any redirect chain. Verify: the final credential page URL is identified.
2. Analyse the URL in an isolated tool such as urlscan or your sandbox, never from a production workstation browser. Verify: screenshots and page behaviour confirm or clear the credential harvest suspicion.
3. If confirmed, block the URL and domain in the tenant list and submit to the gateway vendor for global detection. Verify: the block is active and the submission is acknowledged.
4. Search and purge: find all messages containing the URL or its redirector and remove delivered copies. Verify: the purge count matches the search count.
5. Check for engagement: use audit and proxy logs to see whether anyone submitted credentials, and trigger password resets for anyone who did. Verify: affected users are listed and resets are forced.

Variant phrasings

users reported phishing url gateway missed response

The incident response variant.

extract real url behind link text phishing triage

The extraction variant.

Compatibility: Defender tenant allow block list and submissions, urlscan.io and gateway URL analysis tools.

Why it happens

Reputation feeds are built from reports, so a genuinely new campaign URL has no score anywhere at first delivery. The gateway behaved as designed; the gap is that the tenant had no fast local response loop from user report to block.

Edge cases / pitfalls

• Teach users to use the report button rather than forwarding, which preserves headers and the original URL encoding.
• Redirector URLs rot quickly. Record final domains and page hashes as durable indicators.
• If credentials were submitted, the incident is an account compromise response, not just a mail problem.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_fwnQ8x91TKbnnVnQAGlqvg

Resolved from the public thread: https://vectle.com/posts/pst_fwnQ8x91TKbnnVnQAGlqvg

Thread query: "credential phishing" url flagged by users but missed by gateway

TL;DR: When users find it first, treat the first report as the detection system: extract the real URL from the message, not the screenshot, check it in a safe analysis environment, block it at the tenant, and search for every other recipient. Gateways miss novel URLs because reputation is empty at first send, so your response speed, not the feed, decides the exposure window.