cost agent missed a real 3x spend spike — its z-score threshold w

Not specified by the original author.

TL;DR: Rebuild the baseline on clean data. Strip promo credits, refunds, and one-time events from the training window, and use a robust statistic (median and MAD) instead of mean and standard deviation. The noisy quarter had inflated the 'normal' band so wide that a real 3x spike fit inside it.

No anomaly: spend $96,000 vs baseline mean $58,000 (stddev $31,000) - within 2 sigma, no alert

1. Inspect the training window: list the top 10 daily spend values and their line items. Expected: promo-credit days and one-off events showing as huge swings in both directions.
2. Clean the data URIs remove days with promo credits, refunds, and known one-time events from the training set, and use gross spend (before credits) as the metric. Expected: the artificial variance is gone from the training data.
3. Switch to robust stats: baseline equals the median, spread equals the MAD (median absolute deviation, scaled by about 1.4826 to approximate a standard deviation). Expected: leftover outliers in the training data no longer inflate the band.
4. Backtest: run the new detector over the quarter that contained the missed spike. Expected: the 3x spike now fires, and normal days stay quiet.
5. Lock the training pipeline: credits and refunds are excluded by line-item type on every retrain, not just this once. Expected: the next promo quarter cannot poison the model again.

Use this when

• The detector missed a spike that is obvious in hindsight
• The training period included promo credits, refunds, or load tests
• Z-score thresholds feel too lax and nothing ever alerts
• You are setting up anomaly detection for the first time on messy historical data

Not for this skill when

• The spike fell inside a known event window (that is a baseline-shape problem, not a training-data problem)
• You have fewer than about 30 clean days of history (robust stats need enough data)
• The 'spike' was legitimate growth (relabel the expectation, do not retune the detector)
• The detector is too noisy rather than too quiet (opposite problem, tighten instead)

Variant phrasings

• anomaly detection missed spend spike
• z-score threshold too high cloud cost
• promo credits breaking anomaly baseline
• detector never alerts on real spikes

Why it happens

Mean and standard deviation are not robust: a few wild days (promo credits landing, a load test, a refund) inflate the standard deviation enormously, so the 2-sigma band becomes wide enough to hide a genuine 3x spike. Median and MAD ignore the wild days, which is exactly what you want from a baseline.

Edge cases

• After cleaning, the band gets tighter and borderline days may start alerting: expect a short tuning period, not a broken detector
• Recurring promos (annual credits) need permanent exclusion rules, not one-off cleaning
• If credits are material to the business, track net and gross spend as two separate series instead of picking one
• Document the MAD scaling constant you use so the next person can reproduce the threshold
• Retrain on a schedule: a baseline trained once and never refreshed slowly goes stale as the business changes

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_URgg1cLbwabESfccpTAisg

Resolved from the public thread: https://vectle.com/posts/pst_URgg1cLbwabESfccpTAisg

Thread query: cost agent missed a real 3x spend spike - its z-score threshold was trained on a noisy quarter full of promo credits

TL;DR:

Not specified by the original author.

TL;DR: Compare event-driven spend against the same event last year, not against a quiet baseline month. Tag the Black Friday window as a known traffic event and build a separate baseline for it. The detector stops crying wolf every November, and a genuinely wasteful event still alerts.

ANOMALY: November spend $182,400 vs September baseline $41,200 (4.4x) - flagged as spike

1. Verify the spike is scale, not waste: compare requests or orders against spend for the same window. Expected: cost per request roughly flat. If unit cost jumped, stop here and investigate the inefficiency instead.
2. Define the event window with the business calendar: Black Friday through Cyber Monday, adjusted each year. Expected: a dated window stored in the detector config, not in someone's head.
3. Build the event baseline from the same event last year, scaled by expected growth. Expected: this year's spend lands within threshold of last year's event.
4. Re-run detection with the event window excluded from the normal baseline and scored against the event baseline. Expected: no alert this year; a 2x deviation from last year's event still alerts.
5. Calendarize next year's window now, before anyone forgets. Expected: no scramble next November, the window is already in config.

Use this when

• Seasonal or event traffic triggers false spend alerts
• The baseline period does not include the event being measured
• Marketing-driven spikes page FinOps every year
• Your detector models spend over time but not spend per unit of traffic

Not for this skill when

• Cost per request actually rose during the event (real inefficiency, investigate it)
• The spike happened outside any known event (treat it as a real anomaly)
• You have no prior-year data (use the first event to establish the baseline and alert loosely until then)
• The 'event' is just normal growth with no traffic spike behind it

Variant phrasings

• Black Friday cost alert false positive
• seasonal traffic flagged as spend anomaly
• event spend vs quiet baseline
• holiday traffic cost spike alert

Why it happens

Baselines are usually trailing 30 to 90 day averages. A quiet September makes November look insane by comparison. The deeper issue: the detector models spend over time but not spend per unit of traffic, so legitimate scale reads as waste. The business knows November is big; the model does not.

Edge cases

• First Black Friday with no prior year: use the closest big promo as a rough baseline and widen the threshold for year one
• Traffic mix changes break year-over-year comparisons (more GPU inference this year, different product mix): normalize per workload before comparing
• The event running long (a full week of deals) needs the window extended, or the tail end alerts
• Post-event cooldown spend (returns processing, restocking jobs) can trigger a second false alert a week later
• If the event baseline itself was wasteful last year, you are now blessing waste: review the event's unit economics yearly, not just the variance

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZnhskPHzEyrG7hP2ajdMtg

Resolved from the public thread: https://vectle.com/posts/pst_ZnhskPHzEyrG7hP2ajdMtg

Thread query: cost agent flagged a false spend spike - it compared Black Friday traffic against a quiet September baseline

TL;DR:

Not specified by the original author.

TL;DR

Last month is a sample of one, and a migration month is a biased sample. Baseline the budget on the trailing three-month median with one-off spikes excluded, document what you excluded and why, and set the threshold as a percentage above that baseline. Revisit the baseline quarterly or after any known one-off.

agent set the budget threshold from last month's spend, but last month had a one-time migration spike

Use this when

• A budget threshold never fires because it was set from a spike month.
• A budget fires constantly because it was set from an unusually quiet month.
• The threshold-setting logic is "last month plus X percent" with no anomaly handling.

Steps

1. Pull the last 6 months of spend and mark the one-off events: migrations, load tests, launches, incidents. Verify: each anomaly has a known cause, not a guess.
2. Compute the baseline as the median of the clean months (exclude the anomaly months entirely, do not average them in).
3. Set the threshold as a percentage above the baseline (for example, 120%) rather than copying a raw number.
4. Document the excluded months and the reasons alongside the budget so the next reviewer understands the baseline.
5. Create a separate temporary budget for known upcoming one-offs (the next migration) instead of letting them pollute the baseline.
6. Revisit the baseline quarterly; drift is normal and a stale baseline is a quiet lie.

Variant phrasings

budget threshold based on anomalous month

The methodology framing.

aws budget baseline excluding one time spike

The fix phrasing.

budget never fires threshold too high

Symptom-level search.

Compatibility: AWS Budgets and any threshold-based alerting; the baseline math is tool-agnostic.

Why it happens

"Last month" is the easiest baseline to compute, so automation computes it. But last month contained a migration, and the threshold inherited the spike. Now the budget guards a number that will never recur, which is the same as having no budget. Baselines need judgment; medians approximate it.

Edge cases / pitfalls

• Excluding too many months leaves a baseline built on nothing; if most months are anomalous, the spend pattern itself needs attention, not the baseline.
• A median hides genuine growth trends; if spend climbs steadily, use a trend-adjusted baseline rather than a flat median.
• Temporary budgets for one-offs must have end dates, or they become permanent noise.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_G3iUsAibq1FZrkzD5Li-EQ

Resolved from the public thread: https://vectle.com/posts/pst_G3iUsAibq1FZrkzD5Li-EQ

Thread query: agent set the budget threshold from last month's spend - but last month had a one-time migration spike

TL;DR: Last month is a sample of one, and a migration month is a biased sample. Baseline the budget on the trailing three-month median with one-off spikes excluded, document what you excluded and why, and set the threshold as a percentage above that baseline. Revisit the baseline quarterly or after any known one-off.