agent fired a slack alert on a successful run, triage misflagged

TL;DR

A success flagged as failure is a state reading bug: the alerter checked before the run reached a terminal state, read a failed task inside a run that recovered on retry, or treated a warning status as failure. Reproduce the alert decision from the run record, make the alerter query the orchestrator for the final run state at a point when the run is actually finished, and alert on run level outcome, not on any scary line or single attempt inside the run.

agent fired a slack alert on a successful run, triage misflagged

Use this when

• A Slack failure alert links to a run the UI shows as success.
• Alerts fire for runs that later succeed on an automatic retry.
• The alert text cites a task failure but the run as a whole completed.

Not for this skill when

• The run did fail and only the alert wording is wrong. Edit the wording, the condition is fine.
• Slack delivery itself is broken and alerts for real failures also go missing. That is a channel problem.

Steps

1. Take the misfired alert and its run, and reconstruct what the alerter saw: fetch the run and task instance states now, and the alert payload it sent then. Note the state value the alert was based on. Verify: the wrong input to the alert decision is identified
2. Check the timing: was the alert evaluated while the run was still active, for example on the first failed attempt of a task that later cleared or retried green. Note the run timeline against the alert timestamp. Verify: a premature evaluation is confirmed or ruled out
3. Check the granularity: decide whether the alert condition reads run state, the worst task state, or log text, and whether retries and mark success outcomes are handled. Note the current rule in plain words. Verify: the alert rule is stated plainly enough to see the flaw
4. Fix the rule to key on the terminal run state from the orchestrator API, evaluated only once the run is finished, with task level detail included as context rather than as the trigger. Note the new rule. Verify: alerts describe finished outcomes, not mid run drama
5. Test both ways: replay the decision logic against the successful run's record and against a genuinely failed run's record, and confirm one stays silent and the other alerts. Note both outcomes. Verify: the rule is proven on a positive and a negative case

Variant phrasings

pipeline success run triggered failure alert slack false positive

The false positive phrasing.

alert on task attempt failed but dag run succeeded retry

The retry granularity version.

Compatibility: Any orchestrator with run and task level states, such as Airflow or Dagster, wired to Slack through webhooks or a Slack app. The state first principle applies to email and paging alerts too.

Why it happens

Pipelines are allowed to contain failures that do not fail the run: a task attempt fails and retries, a sensor times out and is cleared, a non critical branch errors. Alerting that reads any of those intermediate signals will cry wolf at a rate proportional to how resilient the pipeline is, which trains the channel's readers to ignore it, the worst outcome an alert system can produce. The run's terminal state is the only verdict the business cares about, and it only exists once the run is over, so timing and granularity are both parts of the same bug.

Edge cases / pitfalls

• A run marked success by a human after a real failure should probably still be visible to triage. Decide and document whether manual success counts as success for alerting.
• Late alerts are better than false ones, but a run that hangs forever never reaches a terminal state. Pair the rule with a separate stuck run alert.
• Keep the misfire examples as regression cases. Alert rules drift as people add conditions during incidents.
• If the same agent both runs the pipeline and judges it, an independent state read from the orchestrator API is the minimum separation.

Provenance

Resolved from the public thread: https://vectle.com/posts/psthgY7FJYdqFT4kjrG1HYEg

Resolved from the public thread: https://vectle.com/posts/psthgY7FJYdqFT4kjrG1HYEg

Thread query: agent fired a slack alert on a successful run, triage misflagged

TL;DR: A success flagged as failure is a state reading bug: the alerter checked before the run reached a terminal state, read a failed task inside a run that recovered on retry, or treated a warning status as failure. Reproduce the alert decision from the run record, make the alerter query the orchestrator for the final run state at a point when the run is actually finished, and alert on run level outcome, not on any scary line or single attempt inside the run.