reported-phish webhook returned 502 and messages never reached the triage agent queue

TL;DR

Make the webhook receiver boring and durable: find why it returns 502, usually an overloaded app, a bad upstream dependency, or a proxy timeout, then put a queue in front of it so the receiver accepts, stores, and acknowledges in milliseconds while the agent processes asynchronously. Add a retry aware response and a dead letter path, and backfill the reports lost during the outage from the submissions portal, because a 502 at report time otherwise deletes the user report from your pipeline forever.

reported-phish webhook returned 502 and messages never reached the triage agent queue

Use this when

• The reporting add-in or portal shows webhook delivery failures with 502 responses.
• Users did report phish, but nothing arrived in the triage agent queue.
• The webhook host logs show restarts, timeouts, or an unhealthy upstream during the same window.

Not for this skill when

• The webhook returns 200 and the agent drops messages later. That is a processing bug downstream.
• Reports were never sent because the add-in was missing for users. That is a deployment problem, not a receiver problem.

Steps

1. Correlate the 502 timestamps with the receiver host metrics and deploy history to find the failing layer: proxy, app, or a dependency it calls synchronously. Work from a copied message or a test send first so production mail stays untouched while you compare the before and after state. Record the timestamp, the sending service, and the exact header or log line that changed, because deliverability and phishing faults often look identical until that evidence is lined up side by side in one place. Verify: the failing layer for the 502 responses is identified with evidence
2. Remove synchronous heavy work from the request path: the receiver should validate, enqueue the raw report, and return success without waiting for analysis. Do not change more than one setting at a time here. If you fix the record, the connector, and the policy in the same window you will not know which change cleared the fault, and the next regression will take just as long to isolate. Save the working configuration once this step checks out before moving on. Verify: a load test shows the receiver responding quickly while analysis is still running
3. Deploy the fix and send test reports through the real user path. Use the receiving side view for this check rather than the sender console alone. Sender dashboards often report a local success while the recipient gateway applied a different verdict, rewrote a header, or dropped the message at a later hop. The recipient header, portal log, or message trace entry is the evidence that counts. Verify: test reports return success and appear in the queue
4. Backfill missed reports by exporting submissions from the portal for the outage window and feeding them through the same queue. Keep any exception you create narrow and time boxed. Scope it to the one sender, domain, or message type you are fixing, note the owner and the review date in the change record, and remove it once the root fix is in place so a temporary workaround does not quietly become a permanent hole in the filter. Verify: the backfilled count matches the portal count for the window
5. Alert on any non success webhook response and on queue depth, and review the alert weekly. Retest with a fresh message through the exact same path as the failing one, not a shortcut path. A new test from a different client, gateway, or mailbox can pass while the original route is still broken, which is the most common reason these faults get closed early and then reopen within a day. Verify: a forced error in test triggers the alert as expected

Why it happens

A 502 means the proxy in front of your receiver could not get a good answer from it, and senders like submission portals treat that as failed delivery. If the receiver does slow work inline, such as fetching the message, calling analysis APIs, and writing a case, every traffic spike or dependency hiccup becomes lost reports. The receiver job is to take custody durably and quickly; everything else belongs behind a queue where retries are safe.

Variants and edge cases

reported-phish webhook returned 502 messages never reached triage queue

The exact status and loss wording.

webhook receiver 502 bad gateway phish reports lost queue fix

The queue architecture variant.

user reported phish not arriving webhook proxy timeout durable receiver

The proxy timeout variant for hosted receivers.

Compatibility: Microsoft 365 user reported message webhooks and submissions APIs, reverse proxy and queue based receiver design.

Edge cases to check:

• Keep the raw report payload in the queue record; reprocessing after a parser fix depends on having the original.
• Agree with the portal sender what it retries and for how long; your backfill window should cover the gap beyond its retries.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ufYVCpdGVcbmezL8mc4CdQ