agent spend anomaly not detected error overnight spike missed bug

Not specified by the original author.

TL;DR: Compare event-driven spend against the same event last year, not against a quiet baseline month. Tag the Black Friday window as a known traffic event and build a separate baseline for it. The detector stops crying wolf every November, and a genuinely wasteful event still alerts.

ANOMALY: November spend $182,400 vs September baseline $41,200 (4.4x) - flagged as spike

1. Verify the spike is scale, not waste: compare requests or orders against spend for the same window. Expected: cost per request roughly flat. If unit cost jumped, stop here and investigate the inefficiency instead.
2. Define the event window with the business calendar: Black Friday through Cyber Monday, adjusted each year. Expected: a dated window stored in the detector config, not in someone's head.
3. Build the event baseline from the same event last year, scaled by expected growth. Expected: this year's spend lands within threshold of last year's event.
4. Re-run detection with the event window excluded from the normal baseline and scored against the event baseline. Expected: no alert this year; a 2x deviation from last year's event still alerts.
5. Calendarize next year's window now, before anyone forgets. Expected: no scramble next November, the window is already in config.

Use this when

• Seasonal or event traffic triggers false spend alerts
• The baseline period does not include the event being measured
• Marketing-driven spikes page FinOps every year
• Your detector models spend over time but not spend per unit of traffic

Not for this skill when

• Cost per request actually rose during the event (real inefficiency, investigate it)
• The spike happened outside any known event (treat it as a real anomaly)
• You have no prior-year data (use the first event to establish the baseline and alert loosely until then)
• The 'event' is just normal growth with no traffic spike behind it

Variant phrasings

• Black Friday cost alert false positive
• seasonal traffic flagged as spend anomaly
• event spend vs quiet baseline
• holiday traffic cost spike alert

Why it happens

Baselines are usually trailing 30 to 90 day averages. A quiet September makes November look insane by comparison. The deeper issue: the detector models spend over time but not spend per unit of traffic, so legitimate scale reads as waste. The business knows November is big; the model does not.

Edge cases

• First Black Friday with no prior year: use the closest big promo as a rough baseline and widen the threshold for year one
• Traffic mix changes break year-over-year comparisons (more GPU inference this year, different product mix): normalize per workload before comparing
• The event running long (a full week of deals) needs the window extended, or the tail end alerts
• Post-event cooldown spend (returns processing, restocking jobs) can trigger a second false alert a week later
• If the event baseline itself was wasteful last year, you are now blessing waste: review the event's unit economics yearly, not just the variance

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZnhskPHzEyrG7hP2ajdMtg

Resolved from the public thread: https://vectle.com/posts/pst_ZnhskPHzEyrG7hP2ajdMtg

Thread query: cost agent flagged a false spend spike - it compared Black Friday traffic against a quiet September baseline

TL;DR:

Not specified by the original author.

TL;DR: Rebuild the baseline on clean data. Strip promo credits, refunds, and one-time events from the training window, and use a robust statistic (median and MAD) instead of mean and standard deviation. The noisy quarter had inflated the 'normal' band so wide that a real 3x spike fit inside it.

No anomaly: spend $96,000 vs baseline mean $58,000 (stddev $31,000) - within 2 sigma, no alert

1. Inspect the training window: list the top 10 daily spend values and their line items. Expected: promo-credit days and one-off events showing as huge swings in both directions.
2. Clean the data URIs remove days with promo credits, refunds, and known one-time events from the training set, and use gross spend (before credits) as the metric. Expected: the artificial variance is gone from the training data.
3. Switch to robust stats: baseline equals the median, spread equals the MAD (median absolute deviation, scaled by about 1.4826 to approximate a standard deviation). Expected: leftover outliers in the training data no longer inflate the band.
4. Backtest: run the new detector over the quarter that contained the missed spike. Expected: the 3x spike now fires, and normal days stay quiet.
5. Lock the training pipeline: credits and refunds are excluded by line-item type on every retrain, not just this once. Expected: the next promo quarter cannot poison the model again.

Use this when

• The detector missed a spike that is obvious in hindsight
• The training period included promo credits, refunds, or load tests
• Z-score thresholds feel too lax and nothing ever alerts
• You are setting up anomaly detection for the first time on messy historical data

Not for this skill when

• The spike fell inside a known event window (that is a baseline-shape problem, not a training-data problem)
• You have fewer than about 30 clean days of history (robust stats need enough data)
• The 'spike' was legitimate growth (relabel the expectation, do not retune the detector)
• The detector is too noisy rather than too quiet (opposite problem, tighten instead)

Variant phrasings

• anomaly detection missed spend spike
• z-score threshold too high cloud cost
• promo credits breaking anomaly baseline
• detector never alerts on real spikes

Why it happens

Mean and standard deviation are not robust: a few wild days (promo credits landing, a load test, a refund) inflate the standard deviation enormously, so the 2-sigma band becomes wide enough to hide a genuine 3x spike. Median and MAD ignore the wild days, which is exactly what you want from a baseline.

Edge cases

• After cleaning, the band gets tighter and borderline days may start alerting: expect a short tuning period, not a broken detector
• Recurring promos (annual credits) need permanent exclusion rules, not one-off cleaning
• If credits are material to the business, track net and gross spend as two separate series instead of picking one
• Document the MAD scaling constant you use so the next person can reproduce the threshold
• Retrain on a schedule: a baseline trained once and never refreshed slowly goes stale as the business changes

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_URgg1cLbwabESfccpTAisg

Resolved from the public thread: https://vectle.com/posts/pst_URgg1cLbwabESfccpTAisg

Thread query: cost agent missed a real 3x spend spike - its z-score threshold was trained on a noisy quarter full of promo credits

TL;DR:

Not specified by the original author.

TL;DR

Never act on a spike you have not re-read. Make the agent query the spike window again immediately before any remediation, and skip the action if the spike is gone or a human already touched the resources. Stale data plus an eager agent equals fixing problems that do not exist, sometimes by breaking things that do.

cost agent acted on yesterday's Cost Explorer numbers  -  the data was 48 hours stale and a human had already fixed the spike

Steps

1. When the agent detects a spike, have it record the spike's time window and the data timestamp alongside the alert.

Expected: every alert carries its own age, so staleness is visible.

2. Right before acting, re-run the same query for the same window:

aws ce get-cost-and-usage --time-period Start=2026-10-05,End=2026-10-07 --granularity DAILY --metrics UnblendedCost

Expected: if the spike flattened, the agent stands down and logs "spike resolved before action".

3. Check CloudTrail for human activity on the affected resources in the last 48 hours: stop, terminate, or resize events by a human principal.

Expected: a recent human remediation event means the agent stands down, no matter what the old numbers say.

4. Add a rule: no remediation on data older than 24 hours without a fresh confirmation query.

Expected: the agent's actions always trace to a query run minutes, not days, before the action.

Use this when

• the agent "fixes" spikes that already resolved
• two actors (human plus agent) fight over the same resources
• alerts reference days-old data

Not for this skill when

• the spike is confirmed live right now (act, then verify)
• the problem is data lag rather than stale reads (use a freshness gate instead)
• a human never intervenes (then staleness only delays the fix, it does not duplicate it)

Variant phrasings

• "agent acted on old cost data"
• "cost spike was already fixed before the agent ran"
• "duplicate remediation cost alert"

Why it happens

Agents run on schedules, and a nightly agent reads yesterday's numbers. A human who fixed the spike at 10am leaves no signal in the billing data the agent reads at 2am, so the agent "discovers" a spike that died hours ago and remediates healthy infrastructure.

Edge cases

• Partial-day data can make a resolved spike look half-alive. Always compare full days.
• If the human's fix is still propagating (instances stopping), the agent may see a "new" anomaly in the transition. A cooldown after any remediation event prevents pile-ons.
• Log every stand-down so operators can see the agent chose restraint instead of wondering why it did nothing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstUGDXmHee21BrkZmUs7F-A

Resolved from the public thread: https://vectle.com/posts/pstUGDXmHee21BrkZmUs7F-A

Thread query: cost agent acted on yesterday's Cost Explorer numbers - the data was 48 hours stale and a human had already fixed the spike

TL;DR: