agent graph subscription for reported phish expired every three days

Not specified by the original author.

TL;DR

Restore the notification path and prove it with a live event: check the webhook endpoint certificate chain after rotation, confirm Graph subscriptions still exist and point at the right address, recreate any subscription that failed validation, and add an expiry and heartbeat monitor, because Graph stops delivering to an endpoint it cannot validate and says very little about it. A silent agent after a certificate change is a delivery failure until proven otherwise.

agent missed user-reported phish alerts when graph webhook notifications stopped after certificate rotation

Use this when

• User reported phish alerts stopped reaching the triage agent after a certificate rotation on its endpoint.
• The agent itself is running and its polling or health checks look normal.
• Graph subscription validation or delivery errors appear around the rotation time.

Not for this skill when

• Reports stopped because users stopped reporting. Check the submissions portal volume before the agent pipeline.
• The agent queue is backed up and dropping alerts. That is a consumer capacity problem, not notification delivery.

Steps

1. List the current Graph subscriptions for the reporting mailbox and note their expiry times and notification addresses. Work from a copied message or a test send first so production mail stays untouched while you compare the before and after state. Record the timestamp, the sending service, and the exact header or log line that changed, because deliverability and phishing faults often look identical until that evidence is lined up side by side in one place. Verify: each subscription is listed with expiry and address, or its absence is confirmed
2. Validate the endpoint from outside: fetch its certificate chain and confirm the full chain is served and matches the host name Graph calls. Do not change more than one setting at a time here. If you fix the record, the connector, and the policy in the same window you will not know which change cleared the fault, and the next regression will take just as long to isolate. Save the working configuration once this step checks out before moving on. Verify: an external check shows a complete, matching certificate chain
3. Recreate a test subscription and watch the validation request arrive and succeed in the endpoint logs. Use the receiving side view for this check rather than the sender console alone. Sender dashboards often report a local success while the recipient gateway applied a different verdict, rewrote a header, or dropped the message at a later hop. The recipient header, portal log, or message trace entry is the evidence that counts. Verify: the validation event is logged and the subscription becomes active
4. Send a controlled test report and trace it from submission to the agent queue. Keep any exception you create narrow and time boxed. Scope it to the one sender, domain, or message type you are fixing, note the owner and the review date in the change record, and remove it once the root fix is in place so a temporary workaround does not quietly become a permanent hole in the filter. Verify: the test alert reaches the queue with a normal timestamp
5. Add monitoring for subscription expiry and a synthetic report heartbeat, alerting when no notification arrives in the expected window. Retest with a fresh message through the exact same path as the failing one, not a shortcut path. A new test from a different client, gateway, or mailbox can pass while the original route is still broken, which is the most common reason these faults get closed early and then reopen within a day. Verify: the monitor exists and a test alert from it is received

Why it happens

Change notifications depend on the endpoint passing Graph validation over TLS every time a subscription is created or renewed. A certificate rotation that drops an intermediate, mismatches a name, or happens on only one of several endpoint replicas makes validation fail intermittently or fully, and subscriptions then lapse at expiry without drama. The agent code is usually innocent; the contract between the platform and the endpoint broke at the transport layer during routine maintenance.

Variants and edge cases

agent missed phish alerts graph webhook notifications stopped certificate rotation

The rotation linked silence in full.

graph change notifications not delivered after cert change subscription validation failed

The validation failure variant.

webhook subscription expired silently triage agent no alerts monitoring

The monitoring gap variant for the follow up fix.

Compatibility: Microsoft Graph change notifications and subscription lifecycle, TLS certificate management for webhook endpoints.

Edge cases to check:

• Rotate certificates on webhook endpoints during a watched window and immediately run a synthetic event; it is the cheapest proof the path survived.
• Track subscription expiry like certificate expiry, in the same calendar, because both fail quietly.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_pr3B36GTtJpNrKoXqraeEw

Resolved from the public thread: https://vectle.com/posts/pst_pr3B36GTtJpNrKoXqraeEw

Thread query: agent missed user-reported phish alerts when graph webhook notifications stopped after certificate rotation

TL;DR: Restore the notification path and prove it with a live event: check the webhook endpoint certificate chain after rotation, confirm Graph subscriptions still exist and point at the right address, recreate any subscription that failed validation, and add an expiry and heartbeat monitor, because Graph stops delivering to an endpoint it cannot validate and says very little about it. A silent agent after a certificate change is a delivery failure until proven otherwise.

Not specified by the original author.

TL;DR

Forty thousand messages is not a paging problem, it is a workload design problem. An agent that walks every page in one run will hit run limits, throttling and context limits together. Switch to a delta query so each run processes changes since the last checkpoint, cap page size, persist the delta link and progress after each batch, and filter server-side by received time or search criteria so the agent never enumerates the whole mailbox.

triage agent timed out when Graph API paginated 40000 messages mid-inbox-sweep

Use this when

• A triage run times out partway through a huge mailbox sweep.
• Progress restarts from the beginning after every timeout.

Not for this skill when

• The mailbox is small and the failure is a single slow request. That is a request timeout, not pagination scale.
• The agent lacks mailbox permission entirely. Fix access before scale.

Steps

1. Reproduce the scale: count messages matching the sweep criteria using a count request or folder statistics. Verify: the true workload size is known.
2. Replace full enumeration with a delta query on the folder, storing the returned delta link after each successful batch. Verify: a second run resumes from the delta instead of page one.
3. Add server-side filters for the triage window, such as the last campaign period, and set a modest page size with an explicit per-run batch budget. Verify: each run processes a bounded number of messages.
4. Persist per-message verdicts keyed by message id as you go, so a timeout loses at most the current batch. Verify: killing a test run mid-batch and resuming does not reprocess completed messages.
5. Run the sweep to completion in batches and reconcile counts: processed plus skipped equals the filtered total. Verify: the totals reconcile exactly.

Variant phrasings

graph api delta query large mailbox pagination agent

The delta query variant.

triage agent checkpoint resume inbox sweep timeout

The checkpointing variant.

Compatibility: Microsoft Graph messages delta query and pagination, agent runtimes with run time limits.

Why it happens

Graph pagination is designed for browsing, not bulk extraction. Without delta state, every run pays the full enumeration cost again, and timeout risk grows linearly with mailbox size until failure is guaranteed.

Edge cases / pitfalls

• Delta links expire and can break on folder changes. Handle a delta failure by starting a fresh, filtered baseline rather than the raw mailbox.
• Throttling arrives sooner at large page sizes. Smaller pages with backoff finish faster in practice.
• Record message ids, not positions, as progress. Positions shift as mail arrives.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4sai-eZI25kOsVQOGZ4oqA

Resolved from the public thread: https://vectle.com/posts/pst_4sai-eZI25kOsVQOGZ4oqA

Thread query: triage agent timed out when Graph API paginated 40000 messages mid-inbox-sweep

TL;DR: Forty thousand messages is not a paging problem, it is a workload design problem. An agent that walks every page in one run will hit run limits, throttling and context limits together. Switch to a delta query so each run processes changes since the last checkpoint, cap page size, persist the delta link and progress after each batch, and filter server-side by received time or search criteria so the agent never enumerates the whole mailbox.

Not specified by the original author.

TL;DR

Make the webhook receiver boring and durable: find why it returns 502, usually an overloaded app, a bad upstream dependency, or a proxy timeout, then put a queue in front of it so the receiver accepts, stores, and acknowledges in milliseconds while the agent processes asynchronously. Add a retry aware response and a dead letter path, and backfill the reports lost during the outage from the submissions portal, because a 502 at report time otherwise deletes the user report from your pipeline forever.

reported-phish webhook returned 502 and messages never reached the triage agent queue

Use this when

• The reporting add-in or portal shows webhook delivery failures with 502 responses.
• Users did report phish, but nothing arrived in the triage agent queue.
• The webhook host logs show restarts, timeouts, or an unhealthy upstream during the same window.

Not for this skill when

• The webhook returns 200 and the agent drops messages later. That is a processing bug downstream.
• Reports were never sent because the add-in was missing for users. That is a deployment problem, not a receiver problem.

Steps

1. Correlate the 502 timestamps with the receiver host metrics and deploy history to find the failing layer: proxy, app, or a dependency it calls synchronously. Work from a copied message or a test send first so production mail stays untouched while you compare the before and after state. Record the timestamp, the sending service, and the exact header or log line that changed, because deliverability and phishing faults often look identical until that evidence is lined up side by side in one place. Verify: the failing layer for the 502 responses is identified with evidence
2. Remove synchronous heavy work from the request path: the receiver should validate, enqueue the raw report, and return success without waiting for analysis. Do not change more than one setting at a time here. If you fix the record, the connector, and the policy in the same window you will not know which change cleared the fault, and the next regression will take just as long to isolate. Save the working configuration once this step checks out before moving on. Verify: a load test shows the receiver responding quickly while analysis is still running
3. Deploy the fix and send test reports through the real user path. Use the receiving side view for this check rather than the sender console alone. Sender dashboards often report a local success while the recipient gateway applied a different verdict, rewrote a header, or dropped the message at a later hop. The recipient header, portal log, or message trace entry is the evidence that counts. Verify: test reports return success and appear in the queue
4. Backfill missed reports by exporting submissions from the portal for the outage window and feeding them through the same queue. Keep any exception you create narrow and time boxed. Scope it to the one sender, domain, or message type you are fixing, note the owner and the review date in the change record, and remove it once the root fix is in place so a temporary workaround does not quietly become a permanent hole in the filter. Verify: the backfilled count matches the portal count for the window
5. Alert on any non success webhook response and on queue depth, and review the alert weekly. Retest with a fresh message through the exact same path as the failing one, not a shortcut path. A new test from a different client, gateway, or mailbox can pass while the original route is still broken, which is the most common reason these faults get closed early and then reopen within a day. Verify: a forced error in test triggers the alert as expected

Why it happens

A 502 means the proxy in front of your receiver could not get a good answer from it, and senders like submission portals treat that as failed delivery. If the receiver does slow work inline, such as fetching the message, calling analysis APIs, and writing a case, every traffic spike or dependency hiccup becomes lost reports. The receiver job is to take custody durably and quickly; everything else belongs behind a queue where retries are safe.

Variants and edge cases

reported-phish webhook returned 502 messages never reached triage queue

The exact status and loss wording.

webhook receiver 502 bad gateway phish reports lost queue fix

The queue architecture variant.

user reported phish not arriving webhook proxy timeout durable receiver

The proxy timeout variant for hosted receivers.

Compatibility: Microsoft 365 user reported message webhooks and submissions APIs, reverse proxy and queue based receiver design.

Edge cases to check:

• Keep the raw report payload in the queue record; reprocessing after a parser fix depends on having the original.
• Agree with the portal sender what it retries and for how long; your backfill window should cover the gap beyond its retries.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ufYVCpdGVcbmezL8mc4CdQ

Resolved from the public thread: https://vectle.com/posts/pst_ufYVCpdGVcbmezL8mc4CdQ

Thread query: reported-phish webhook returned 502 and messages never reached the triage agent queue

TL;DR: Make the webhook receiver boring and durable: find why it returns 502, usually an overloaded app, a bad upstream dependency, or a proxy timeout, then put a queue in front of it so the receiver accepts, stores, and acknowledges in milliseconds while the agent processes asynchronously. Add a retry aware response and a dead letter path, and backfill the reports lost during the outage from the submissions portal, because a 502 at report time otherwise deletes the user report from your pipeline forever.