CamelCamelCamel alert not firing on price drop: alert threshold error fix

Not specified by the original author.

TL;DR

Review SLO queries like code. - Alerting on the SLI directly instead of burn rate: you get paged for blips that consume 0.1 percent of budget. - Too many SLOs. Fastest path, inherits the monitor quirks (fix the monitor first). - Time-slice SLO: good/bad minutes from metrics. Target: 99, 99.5, 99.9, pick from the business, not from ambition. Each nine roughly 10xes the engineering cost.

When to use

You are seeing this: Most flexible, most work: you write the numerator/denominator queries. - Monitor-based SLO: you already have a monitor that defines bad. Use this skill when you run into "SLO setup: pick the type, set burn-rate alerts, test them".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

SLO setup that survives contact with reality

Pick the type

• Metric SLO: you have good/bad event counts as metrics (request counts, error counts). Most flexible, most work: you write the numerator/denominator queries.
• Monitor-based SLO: you already have a monitor that defines bad. Fastest path, inherits the monitor quirks (fix the monitor first).
• Time-slice SLO: good/bad minutes from metrics. Best for availability-style objectives with clear per-minute verdicts.

The setup flow

1. Target: 99, 99.5, 99.9, pick from the business, not from ambition. Each nine roughly 10xes the engineering cost.
2. Window: 7, 30, or 90 days rolling. Shorter windows react faster and noisily; 30d is the sane default.
3. Name, description, tags: name it for the user experience it protects, describe why it matters, tag team and service so the SLO list is searchable.
4. Thresholds: target plus a warning threshold above it (warning fires before the objective is breached).
5. Burn-rate alerts: alert on budget consumption rate, not on the raw error ratio. A fast-burn alert (1h window) pages; a slow-burn alert (6h+) tickets. Tune the windows to your traffic: low-traffic services need longer windows or one bad minute pages you.

Traps

• SLO on a metric that changes tags next quarter. The query silently stops matching and the SLO goes green forever. Review SLO queries like code.
• Alerting on the SLI directly instead of burn rate: you get paged for blips that consume 0.1 percent of budget.
• Too many SLOs. Five SLOs per service that nobody looks at lose to one SLO the team actually defends in review.

Verify

Burn a little budget on purpose in staging (synthetic errors) and watch the fast-burn alert fire and the SLO view reflect it. An SLO whose alert you have never seen fire is untested.

The SLO API docs define the allowed types (metric, monitor, time_slice) with thresholds, timeframes (7d/30d/90d), and targets; the SLO checklist guide walks through target selection (99, 99.5, 99.9), time windows, naming, description, and team/service tags. Verified against live docs 2026-09-26.

Not specified by the original author.

TL;DR

Likely causes: (a) alert window too short for the traffic pattern (low-traffic service, one bad minute burns the fast window), (b) SLO query changed (new tags split the good/bad events differently), (c) the underlying monitor feeding a monitor-based SLO flaps, (d) time zone / window alignment confusion.

Steps

1. Symptom (exact): burn-rate alert pages, but the service looks healthy and the error budget looks intact in the SLO view.
2. Open the SLO: what is the actual budget remaining? If it is healthy, the alert math disagrees with the SLO math, which points at (a) or (b).
3. Look at the alert evaluation: which window burned? A 1h window burning on a service with 10 requests/hour is noise, not signal.
4. Diff the SLO query against last week. A tag rename or a new status code in the bad-events filter changes the ratio without changing reality.
5. For monitor-based SLOs: is the underlying monitor flapping on no-data? Fix the monitor first.
6. Fix: lengthen the fast-burn window for low-traffic services, fix the query, or split the SLO per traffic class. Do not just raise the threshold: that hides the next real breach.
7. Verify: the alert stays quiet through a normal traffic cycle and fires on a synthetic error burst (test with a canary if you can). An alert you have never seen fire correctly is a decoration.

When to use

You are seeing this: Symptom (exact): burn-rate alert pages, but the service looks healthy and the error budget looks intact in the SLO view. Use this skill when you run into "Diag: burn-rate alert misfiring, check window math before thresholds".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Datadog SLOs support time-slice and monitor-based types with burn-rate alerting; a misfiring burn alert is usually window math, not a real budget breach. Verified against live docs 2026-09-26.

Not specified by the original author.

TL;DR

Fix this in health score alert by acting on the exact message first: test the alert rule against one account already below the threshold, check the rule compares the current overall score and not a single measure, confirm the rule schedule and the entry condition are both active, and fix a threshold set on the wrong scale, for example 0.7 against a 0 to 100 score. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one account, batch or record set first, then rerun the full scope and reconcile the counts.

health score alert not triggering when account falls below threshold

Use this when

• The operation in health score alert fails with this exact message or symptom: health score alert not triggering when account falls below threshold.
• The same failure repeats on retry with unchanged inputs.
• Scoring, alerting or routing is blocked until this item completes.

Not for this skill when

• The message is an outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output quality. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in health score alert, and copy the full message plus the account, measure or job involved into the run notes. Verify: the error is reproduced on demand with a named scope, not only seen in a log.
2. Apply the specific fix for this error: test the alert rule against one account already below the threshold, check the rule compares the current overall score and not a single measure, confirm the rule schedule and the entry condition are both active, and fix a threshold set on the wrong scale, for example 0.7 against a 0 to 100 score. Verify: the smallest failing scope now completes, or the named account, measure or job behaves as the fix predicts.
3. Check the neighbouring accounts, measures or jobs that share the same config, schema or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in health score alert on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts, scores or status against the source in health score alert so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

Alert rules fail silently when the threshold scale, the score field, or the evaluation schedule does not match the score they watch, so accounts cross the line with no rule ever firing.

Variant phrasings

health score alert not triggering when account falls below threshold

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single scope proof, then the full rerun.

How to fix - health score alert not triggering when account falls below threshold error failure

The how-to phrasing for the same failure in health score alert. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

health score alert troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Edge cases

• If the fix works on the sample scope but the full rerun still fails, a second account or record with the same fault is present; widen the scope in stages until the next failing item is found.
• If scores change after the fix but downstream alerts or reports do not, check the downstream refresh timing before recalculating again, because those surfaces often read a synced copy.
• If the source data itself is missing for the affected accounts, record that as a data gap rather than scoring the accounts as zero.

Compatibility: health score alert at current versions, plus the platform API or console the query names; no extra tooling beyond the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JffAqtjP1mDaud2tGiqfGA

Resolved from the public thread: https://vectle.com/posts/pst_JffAqtjP1mDaud2tGiqfGA

Thread query: health score alert not triggering when account falls below threshold error failure

TL;DR: Fix this in health score alert by acting on the exact message first: test the alert rule against one account already below the threshold, check the rule compares the current overall score and not a single measure, confirm the rule schedule and the entry condition are both active, and fix a threshold set on the wrong scale, for example 0.7 against a 0 to 100 score. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one account, batch or record set first, then rerun the full scope and reconcile the counts.