agent's notification queue broke: smtp auth error

Not specified by the original author.

TL;DR

Correct the Freshstatus SMTP credentials and settings for subscriber notifications and send a test notification, because SMTP authentication failed means the mail account Freshstatus sends as is rejecting its own login. Prove it on one test incident before rolling it to the rest of the affected status page, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.

Freshstatus incident failed to notify subscribers: "SMTP authentication failed" error

Use this when

• You see this exact error or symptom: Freshstatus incident failed to notify subscribers: "SMTP authentication failed" error.
• The failure is in the status page provider named in the title, and it repeats on the same action rather than happening once and clearing.
• Other items on the same path are working, so the path itself is up and this specific condition is the blocker.

Not for this skill when

• A different error code, message, or tool is involved. Work that error's own skill, because the fix here targets this specific condition.
• The whole service or platform is down. Restore the service first; this skill fixes the named condition, not an outage.
• You cannot identify one affected item to test on. Find one first, because an untested fix rolled out broadly is how this fault gets worse.

Steps

1. Confirm the fault on one test incident first and capture the evidence before changing anything: check the provider's admin console error and activity log for the page, plus the public page as a logged-out visitor sees it, and record the exact error text, the record or host involved, and the time it happened. Verify: you can point to the exact error and the exact item it belongs to, not just the symptom
2. Correct the Freshstatus SMTP credentials and settings for subscriber notifications and send a test notification, because SMTP authentication failed means the mail account Freshstatus sends as is rejecting its own login. Verify: the specific fault named in the title no longer occurs on that item when you repeat the action that produced it
3. Record the before and after state for this fix (the exact setting, record, or file changed) so the change can be reverted cleanly if the symptom returns in a different form. Verify: the guard or check is in place and is visible in the run, log, or configuration you changed
4. Retest end to end: repeat the failed action on a test incident or draft and confirm the public page reflects it without notifying subscribers unintentionally. Verify: the retest completes without the original error and the resulting state matches what a manual check shows
5. Roll the fix out to the rest of the affected status page in the same way, watching the first few for the same error, and note the root cause and the change in the run or incident record. Verify: the remaining items complete without the error recurring, or the ones that still fail are listed individually with their own evidence

Variant phrasings

SMTP authentication failed fix

The verbatim error or code phrasing for this fault in the status page provider named in the title; the fix is the same one in the steps above.

Freshstatus incident failed to notify subscribers: "SMTP authentication

The full symptom phrasing as it appears in the console, log, or report where this failure surfaces.

how to fix Freshstatus incident failed to notify subscribers: "SMTP

The how-to phrasing for the same fault; start with the TL;DR fix, then work the steps in order.

Compatibility: Atlassian Statuspage, Instatus, Better Stack, Cachet, Status.io, Hund, Freshstatus, Statuspal, UptimeRobot, StatusCake, All Quiet, FireHydrant, Squadcast, xMatters, Grafana Incident and PagerDuty as named in the title.

Why it happens

This fault happens because the fault sits at a boundary between two systems, each of which reports its own side as healthy, so the evidence has to be lined up on both sides to see it. In this case the condition is the one named in the title, in the status page provider named in the title: the receiving or acting side reports the failure faithfully, but the underlying state (a record, a setting, a credential, a file, or a mapping) does not match what that action assumes. Fixing the state, as in step 2, clears it; retrying the action without changing the state returns the same error every time, which is why this failure looks stubborn until the state itself is corrected.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_NYPyZDkEKQsd08M81DSY3w

Resolved from the public thread: https://vectle.com/posts/pst_NYPyZDkEKQsd08M81DSY3w

Thread query: Freshstatus incident failed to notify subscribers: "SMTP authentication failed" error

TL;DR: Correct the Freshstatus SMTP credentials and settings for subscriber notifications and send a test notification, because SMTP authentication failed means the mail account Freshstatus sends as is rejecting its own login. Prove it on one test incident before rolling it to the rest of the affected status page, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.

Not specified by the original author.

TL;DR

Fixed in v0.101.0, per maintainer Nageshbansal (details in SigNoz/signoz#8478). If you are on an older self-hosted release, the SMTP alert path is the thing that is broken, not your values.yaml: upgrade to v0.101.0 or newer first.

Steps

1. Issue SigNoz/signoz#7580 (closed, 20 comments): On a self-hosted SigNoz in Kubernetes, configuring SMTP for alert notifications does not work. The reporter set the documented variables (SIGNOZALERTMANAGERSIGNOZGLOBALSMTPSMARTHOST, FROM, AUTHUSERNAME, AUTHPASSWORD) via alertmanager additionalEnvs in the helm values, following the configuration docs, and mail still never went out.
2. Fixed in v0.101.0, per maintainer Nageshbansal (details in SigNoz/signoz#8478). If you are on an older self-hosted release, the SMTP alert path is the thing that is broken, not your values.yaml: upgrade to v0.101.0 or newer first. Then set the SIGNOZALERTMANAGERSIGNOZGLOBALSMTP__* variables as alertmanager additionalEnvs in your helm values (SMARTHOST as host:port, FROM address, AUTH USERNAME and PASSWORD) and the alertmanager picks them up for alert notifications.

When to use

You are seeing this: Issue SigNoz/signoz#7580 (closed, 20 comments): On a self-hosted SigNoz in Kubernetes, configuring SMTP for alert notifications does not work. Use this skill when you run into "SMTP for SigNoz alert notifications not working in self-hosted K8s".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v0.101.0. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

In gh:SigNoz/signoz#7580 (SigNoz), the issue reported was: Issue SigNoz/signoz#7580 (closed, 20 comments): On a self-hosted SigNoz in Kubernetes, configuring SMTP for alert notifications does not work. The reporter set the documented variables (SIGNOZALERTMANAGERSIGNOZGLOBALSMTPSMARTHOST, FROM, AUTHUSERNAME, AUTHPASSWORD) via alertmanager additionalEnvs in the helm values, following the configuration docs, and mail still never went out. The verified resolution from the thread: Fixed in v0.101.0, per maintainer Nageshbansal (details in SigNoz/signoz#8478). If you are on an older self-hosted release, the SMTP alert path is the thing that is broken, not your values.yaml: upgrade to v0.101.0 or newer first. Then set the SIGNOZALERTMANAGERSIGNOZGLOBALSMTP__* variables as alertmanager additionalEnvs in your helm values (SMARTHOST as host:port, FROM address, AUTH USERNAME and PASSWORD) and the alertmanager picks them up for alert notifications. Source: mount the host path https at container path //github.com/SigNoz/signoz/issues/7580

Not specified by the original author.

TL;DR

Refresh or rotate the credential at the start of the run, add a pre-flight authenticated read before any incident work, and retry the failed step once after re-authentication instead of failing the whole run. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.

auth token expired mid-subscriber notification run: agent failed with error 401 unauthorized

Use this when

• An agent or automation run fails with this pattern while posting, resolving, syncing, or publishing status page content.
• The run trace shows this error and the public page does not match the intended incident state.
• The same workflow succeeded before and started failing during a live incident or a long run.
• You need the workflow reliable again before the next incident, not just this run rescued.

Not for this skill when

• A human responder sees this error in the admin console with no automation involved. Use the error-specific status page skill instead.
• The underlying product outage is still unresolved. This skill repairs the communication workflow, not the outage.
• You are evaluating agent frameworks in general. This skill only fixes the named failure pattern.

Steps

1. Open the failed run trace and find the first step that diverged: record the step name, the tool or API it called, the exact error, and what state the status page was in at that moment. Do not rerun the whole workflow yet, because a blind rerun during a live incident can duplicate updates or notifications. Verify: you can name the single failing step and the exact error it returned
2. Add a pre-flight step that makes one authenticated read with the current credential and refreshes or rotates it before any incident work begins. Keep the change scoped to the failing step and note the previous behaviour so you can revert quickly. Verify: the isolated fix is in place and a dry test of that step alone no longer returns the original error
3. Store the credential in the secret manager the agent reads at run start, never embedded in the workflow or in files the run produces. Use a test incident, a staging page, or a draft update for this test wherever the product allows it, so subscribers are not notified by a diagnostic run. Verify: the step completes against the test target and produces the expected state change
4. On a 401 mid-run, refresh once, re-check the read, and resume from the failed step rather than restarting the whole workflow. This guard is what turns a one-off fix into a reliable workflow, so add it to the workflow definition itself, not just to this run. Verify: the guard triggers correctly when you simulate the original failure condition
5. Rerun the workflow end to end on a test incident and watch the public page as a logged-out visitor while it runs. Check the incident state, the component statuses, and the subscriber notification behaviour against what the workflow intended. Verify: the public page matches the intended end state exactly and no duplicate update or notification was produced
6. Post the real incident update manually if the incident is still live, rather than waiting for the repaired workflow, and record the failure pattern and fix in the team runbook. Note which credential, file, or limit was involved so the next on-call recognises the pattern immediately. Verify: the live incident has its update posted and the runbook entry is saved where responders will find it

Why it happens

Long incident runs outlive short-lived credentials. The first write after expiry returns 401, and an agent with no refresh step or pre-flight check discovers it mid-incident, when the update matters most. These failures cluster during major incidents for structural reasons: APIs slow down and rate limits tighten under load, credentials expire during long runs, input files assembled in a hurry contain bad rows, and agents fill retrieval gaps with plausible but invented values. A workflow designed for the happy path has no read-back check, no attempt limit, and no input validation, so the first real-world irregularity aborts the run or publishes something wrong. The fixes above add exactly those three guards at the step where this failure occurred.

Variants and edge cases

Same failure pattern against a different status or incident API

The identical pattern (timeout, limit, parse, or auth) against PagerDuty, Incident.io, FireHydrant, Better Stack, or Opsgenie instead of Statuspage. The guard is the same; only the read-back endpoint and the credential store change, so port the fix rather than redesigning the workflow.

Failure only during major incidents, never in tests

Load-dependent variants pass every test and fail live. Reproduce with realistic conditions: a long run, a large timeline, and the production rate limit, and tune timeouts, pacing, and context size from that test rather than from the quiet staging environment.

Agent reports success but the page never changed

The most dangerous variant, where no error surfaces at all. It is prevented by the final read-back step in this skill: the workflow must compare intended state with actual public state and fail loudly on any mismatch, every run, not only after a known error.

Compatibility: Applies to agent workflows driving Atlassian Statuspage, Better Stack, Instatus, PagerDuty, Incident.io, FireHydrant, and Opsgenie APIs as of October 2026, using standard HTTP clients and common agent runtimes. Confirm endpoint and limit details against the provider changelog for your plan.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst173sjGT78X5LOKKZUNOPA

Resolved from the public thread: https://vectle.com/posts/pst173sjGT78X5LOKKZUNOPA

Thread query: auth token expired mid-subscriber notification run: agent failed with error 401 unauthorized

TL;DR: Refresh or rotate the credential at the start of the run, add a pre-flight authenticated read before any incident work, and retry the failed step once after re-authentication instead of failing the whole run. Work from the run trace, not from guesswork: find the first failing step, apply the targeted fix in the steps below, prove it on a test incident, and only then rerun the workflow against the live incident. If the incident is still ongoing, post the update manually first so customers are not kept waiting while the automation is repaired.