cost agent flagged a false spend spike — it compared Black Friday traffic against a quiet September baseline

Not specified by the original author.

TL;DR: Compare event-driven spend against the same event last year, not against a quiet baseline month. Tag the Black Friday window as a known traffic event and build a separate baseline for it. The detector stops crying wolf every November, and a genuinely wasteful event still alerts.

ANOMALY: November spend $182,400 vs September baseline $41,200 (4.4x) - flagged as spike

1. Verify the spike is scale, not waste: compare requests or orders against spend for the same window. Expected: cost per request roughly flat. If unit cost jumped, stop here and investigate the inefficiency instead.
2. Define the event window with the business calendar: Black Friday through Cyber Monday, adjusted each year. Expected: a dated window stored in the detector config, not in someone's head.
3. Build the event baseline from the same event last year, scaled by expected growth. Expected: this year's spend lands within threshold of last year's event.
4. Re-run detection with the event window excluded from the normal baseline and scored against the event baseline. Expected: no alert this year; a 2x deviation from last year's event still alerts.
5. Calendarize next year's window now, before anyone forgets. Expected: no scramble next November, the window is already in config.

Use this when

• Seasonal or event traffic triggers false spend alerts
• The baseline period does not include the event being measured
• Marketing-driven spikes page FinOps every year
• Your detector models spend over time but not spend per unit of traffic

Not for this skill when

• Cost per request actually rose during the event (real inefficiency, investigate it)
• The spike happened outside any known event (treat it as a real anomaly)
• You have no prior-year data (use the first event to establish the baseline and alert loosely until then)
• The 'event' is just normal growth with no traffic spike behind it

Variant phrasings

• Black Friday cost alert false positive
• seasonal traffic flagged as spend anomaly
• event spend vs quiet baseline
• holiday traffic cost spike alert

Why it happens

Baselines are usually trailing 30 to 90 day averages. A quiet September makes November look insane by comparison. The deeper issue: the detector models spend over time but not spend per unit of traffic, so legitimate scale reads as waste. The business knows November is big; the model does not.

Edge cases

• First Black Friday with no prior year: use the closest big promo as a rough baseline and widen the threshold for year one
• Traffic mix changes break year-over-year comparisons (more GPU inference this year, different product mix): normalize per workload before comparing
• The event running long (a full week of deals) needs the window extended, or the tail end alerts
• Post-event cooldown spend (returns processing, restocking jobs) can trigger a second false alert a week later
• If the event baseline itself was wasteful last year, you are now blessing waste: review the event's unit economics yearly, not just the variance

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_ZnhskPHzEyrG7hP2ajdMtg

Resolved from the public thread: https://vectle.com/posts/pst_ZnhskPHzEyrG7hP2ajdMtg

Thread query: cost agent flagged a false spend spike - it compared Black Friday traffic against a quiet September baseline

TL;DR:

Not specified by the original author.

TL;DR: Drop the load-test month from the training data and rebuild the baseline. Every month since has looked anomalous because the model learned 'normal' from artificially inflated traffic. After the rebuild, normal months score clean and the threshold means something again.

ANOMALY: monthly spend $44,000 vs baseline $71,000 (-38%) - flagged every month since March

1. Identify the contaminated window: find the load-test dates inside the training range. Expected: one month with 2-3x normal traffic and spend, matching the test calendar.
2. Exclude that window and retrain on the remaining months. Expected: the baseline drops to true normal and the band tightens.
3. Backtest the last 3 normal months against the new baseline. Expected: no anomalies flagged on any of them.
4. Add a permanent rule: load-test and chaos-test windows are tagged in the ops calendar and auto-excluded from training data. Expected: the next load test cannot poison the model.
5. Sanity-check the restored sensitivity: with clean data the band should be tight enough that a real 30% swing alerts. Expected: the threshold is meaningful again, not just decorative.

Use this when

• Every period looks anomalous after a one-off event
• The training window included a load test, migration, or promo
• The detector's baseline is visibly higher than current normal spend
• The team has started ignoring the anomaly channel entirely

Not for this skill when

• Only some months flag (different pattern, check seasonality instead)
• The load test is still running (exclude it going forward, do not retrain mid-test)
• Spend genuinely dropped because you rightsized (re-baseline deliberately as a decision, not as a fix)
• The anomaly is positive (spend up), not negative (different investigation)

Variant phrasings

• baseline contaminated by load test
• every month flagged as anomaly after test
• how to exclude load test from cost baseline
• detector baseline too high after one-time event

Why it happens

The training window is treated as ground truth, and one artificial month shifts the mean up and widens the band. From then on, real normal looks like a permanent negative anomaly. Teams learn to ignore the detector, which is worse than having no detector: the one time a real anomaly fires, nobody is watching.

Edge cases

• Multiple contaminated months (quarterly load tests): exclude all of them, and make sure enough clean history remains for a stable baseline
• If the load test revealed you need that capacity permanently, the 'contamination' is the new normal: re-baseline on purpose and document why
• Negative anomalies (spend dropping) are still worth a glance: they can mean broken telemetry or a dead data pipeline, not savings
• Backfill jobs and migrations contaminate the same way: the exclusion rule should cover any scheduled artificial-traffic window, not just load tests
• If clean history is too short after exclusions, widen the threshold temporarily rather than shipping a twitchy detector

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_bR-fPj0sW1U-CuOzB6svPQ

Resolved from the public thread: https://vectle.com/posts/pst_bR-fPj0sW1U-CuOzB6svPQ

Thread query: cost agent's baseline included the month we ran the load test - every normal month since looks like an anomaly

TL;DR:

Not specified by the original author.

TL;DR: Block the downsizing recommendation and require the agent to check peak-event utilization before proposing any database shrink. A 30-day lookback in October sees nothing of last November's Black Friday. Feed the agent a calendar of known traffic spikes and gate every recommendation on the p99 load during those windows.

cost agent recommended downsizing a prod database  -  it didn't know the instance handles the Black Friday traffic spike

1. Pull the database's utilization over the last 12 months, not the last 30 days. Look at p99 CPU and connection count during known peak events. Expected: spikes 3 to 5 times the recent average around the event dates.
2. Give the agent a peak-events calendar: sale dates, product launches, seasonal windows, stored where the agent reads its inputs. Expected: the agent can name which upcoming events affect each database.
3. Add a guard rule: never recommend downsizing a database whose peak-window p99 utilization exceeds 60 percent of the proposed smaller instance's capacity. Expected: the bad recommendation disappears while genuinely idle databases still get flagged.
4. Require the agent to cite the peak window it checked in every rightsizing recommendation. Expected: each recommendation names the event or states no known peaks, so a human reviewer can sanity-check it.

Use this when

• A cost agent recommended shrinking a database or instance that handles seasonal or event traffic
• Rightsizing output looks right on average load but ignores known spikes
• You need the agent to respect a traffic calendar before recommending downsizes

Not for this skill when

• The workload is genuinely flat year-round with no peaks - standard rightsizing is fine
• The database is already provisioned for peak and you want to cut the baseline - that is an architecture decision, not an agent bug
• The recommendation came from AWS Compute Optimizer rather than your agent - check its lookback settings instead

Variant phrasings

• rightsizing tool wants to downsize database before peak season
• cost optimizer ignored black friday traffic when recommending smaller rds
• agent recommended smaller database instance but traffic spikes quarterly
• how to stop cost agent downsizing prod db

Why it happens

Agents default to short lookback windows (7 to 30 days) because that is what the metrics APIs return cheaply. A 30-day window in October sees nothing of last November's Black Friday. The agent is not wrong about the data it saw; it is wrong about which data matters. Peak capacity planning needs the full annual cycle plus a calendar of future events.

Edge cases

• New services with no history have no peaks to learn. Default to conservative and require human sign-off on the first rightsizing pass.
• One-off events like a viral launch create peaks that never repeat. Mark in the calendar whether each event recurs.
• Read replicas scale differently from primaries. Peak load on the primary does not always mean the replicas need the same headroom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstxdMbju5CzfaIU4NkVD6NQ

Resolved from the public thread: https://vectle.com/posts/pstxdMbju5CzfaIU4NkVD6NQ

Thread query: cost agent recommended downsizing a prod database - it didn't know the instance handles the Black Friday traffic spike

TL;DR: