agent's anomaly detector kept paging on Sundays - it never modeled the weekend batch ETL window

TL;DR: Give the detector a day-of-week feature, or simply keep a separate weekend baseline. The Sunday ETL is expected spend; the fix is comparing Sundays to Sundays instead of to weekdays. One config change and the Sunday pages stop.

ANOMALY: Sunday spend $2,840 vs expected $420 (6.8x baseline) - paging on-call (3rd Sunday in a row)

1. Confirm the weekly pattern: group daily spend by day of week for the last 8 weeks. Expected: Sundays consistently elevated, weekdays flat.
2. Identify the job: check what runs on Sundays (ETL, backups, weekly reports). Expected: a named job with an owner and an expected cost.
3. Split the baseline: one model for weekdays, one for weekends, or add day-of-week as a model feature. Expected: Sunday spend is scored against prior Sundays.
4. Verify with two Sundays of clean runs, then sanity-check the guardrail: a Sunday at 2x the normal Sunday cost should still alert. Expected: quiet Sundays, loud broken Sundays.
5. Document the ETL window in the runbook like any other known batch window: owner, expected cost, expected duration. Expected: the on-call entry exists before the next person needs it.

Use this when

• Alerts fire on the same weekday every week
• Weekend batch jobs page on-call on a schedule
• The detector has no day-of-week awareness
• Spend grouped by weekday shows a clear weekend pattern

Not for this skill when

• The weekend spike is new with no history (investigate first, it might be real)
• Weekday spend is also spiking (broader problem, not a weekly pattern)
• The job runs at irregular times (use an event-driven window, not day-of-week)
• The ETL cost is growing week over week (real trend inside the window, investigate the job)

Variant phrasings

• anomaly alert every Sunday
• weekend batch job false positive cost alert
• day of week spend pattern not modeled
• detector pages on weekends only

Why it happens

Same root as the monthly-batch problem: a single rolling baseline cannot learn weekly seasonality. Sunday ETL spend looks like a 7x outlier against a baseline built mostly from Monday-to-Friday. Without a day-of-week feature the model is structurally blind to the most predictable pattern in the data.

Edge cases

• Holidays that fall on weekdays look like weekends to the model: add a holiday calendar or accept the occasional false alert
• The ETL occasionally runs Saturday instead of Sunday: allow adjacent-day tolerance, or trigger the window off the job's completion event
• ETL cost drifting up 10% month over month is a real trend hiding inside the 'normal' window: track the Sunday average as its own series
• Timezone: 'Sunday' in UTC versus the team's timezone can split the window across two days. Pick one timezone and be consistent
• If the ETL moves to weekdays one day, remember to remove the weekend model or it will mask real weekend anomalies

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gy5bSNQ9M7Q1rOByMuY31Q