anomaly detector fires on every scheduled batch job window and misses real incidents failure

Not specified by the original author.

TL;DR

Fix this in the anomaly detector by acting on the exact message first: mark the scheduled batch windows as known events with their own expected range, or exclude those windows from the baseline, and judge the detector on incidents outside batch windows for a week. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

anomaly detector fires on every scheduled batch job window and misses real incidents failure

Use this when

• The operation in the anomaly detector fails with this exact message: anomaly detector fires on every scheduled batch job window and misses real incidents failure.
• The same failure repeats on retry with unchanged inputs.
• A review, export or check is blocked until this item completes.

Not for this skill when

• The message is a permission or outage notice from the platform. Resolve that first, then return here if this error remains.
• The operation completes and you only disagree with its output. That is a quality review, not this failure.

Steps

1. Reproduce the failure with the smallest scope that still shows it in the anomaly detector, and copy the full message plus the item, user or metric involved into the run notes. Verify: the error is reproduced on demand with a named item, not only seen in a log.
2. Apply the specific fix for this error: mark the scheduled batch windows as known events with their own expected range, or exclude those windows from the baseline, and judge the detector on incidents outside batch windows for a week. Verify: batch windows stop alerting and a test incident outside them still pages.
3. Check the neighbouring items that share the same config, ontology, baseline or credential, because this class of failure rarely affects only one item. Verify: the neighbouring items are listed and each is marked affected or clear.
4. Rerun the original operation in the anomaly detector on the previously failing scope, watching for the same message rather than assuming success from a quiet log. Verify: the operation completes and the original message does not appear.
5. Compare the result counts, decisions or alerts against the source in the anomaly detector so a silent partial success is caught before sign off. Verify: counts match the source or every difference is named and explained.
6. Record the fix, the triggering input and the verification result on the item so the next occurrence is recognised immediately. Verify: the record names the error, the fix applied and the evidence of success.

Why it happens

A detector trained on quiet periods treats every predictable batch surge as an anomaly, and the noise hides the incidents that matter.

Variant phrasings

anomaly detector fires on every scheduled batch job window and misses real incidents failure

The verbatim form searchers paste from the terminal or console. Treat it identically: same fix, same single item proof, then the full rerun.

How to fix: anomaly detector fires on every scheduled batch job window and misses real incidents failure

The how-to phrasing for the same failure in the anomaly detector. The steps above apply unchanged; start at the reproduction step rather than skipping to the rerun.

the anomaly detector troubleshooting for this failure

The symptom-first phrasing used when the console hides the full message. Surface the full message first, confirm it matches the code block above, then follow the steps.

Compatibility: the anomaly detector at current versions, plus the platform API or console the query names; no extra tooling beyond a text editor and the source system access needed to verify counts.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_KWKHTbwxYwqDfunf0klkFA

Resolved from the public thread: https://vectle.com/posts/pst_KWKHTbwxYwqDfunf0klkFA

Thread query: anomaly detector fires on every scheduled batch job window and misses real incidents failure

TL;DR: Fix this in the anomaly detector by acting on the exact message first: mark the scheduled batch windows as known events with their own expected range, or exclude those windows from the baseline, and judge the detector on incidents outside batch windows for a week. Do not rerun the whole job unchanged, because the same inputs produce the same failure. Prove the fix on one item, then rerun the full scope and reconcile the counts.

Not specified by the original author.

TL;DR: Give the detector a day-of-week feature, or simply keep a separate weekend baseline. The Sunday ETL is expected spend; the fix is comparing Sundays to Sundays instead of to weekdays. One config change and the Sunday pages stop.

ANOMALY: Sunday spend $2,840 vs expected $420 (6.8x baseline) - paging on-call (3rd Sunday in a row)

1. Confirm the weekly pattern: group daily spend by day of week for the last 8 weeks. Expected: Sundays consistently elevated, weekdays flat.
2. Identify the job: check what runs on Sundays (ETL, backups, weekly reports). Expected: a named job with an owner and an expected cost.
3. Split the baseline: one model for weekdays, one for weekends, or add day-of-week as a model feature. Expected: Sunday spend is scored against prior Sundays.
4. Verify with two Sundays of clean runs, then sanity-check the guardrail: a Sunday at 2x the normal Sunday cost should still alert. Expected: quiet Sundays, loud broken Sundays.
5. Document the ETL window in the runbook like any other known batch window: owner, expected cost, expected duration. Expected: the on-call entry exists before the next person needs it.

Use this when

• Alerts fire on the same weekday every week
• Weekend batch jobs page on-call on a schedule
• The detector has no day-of-week awareness
• Spend grouped by weekday shows a clear weekend pattern

Not for this skill when

• The weekend spike is new with no history (investigate first, it might be real)
• Weekday spend is also spiking (broader problem, not a weekly pattern)
• The job runs at irregular times (use an event-driven window, not day-of-week)
• The ETL cost is growing week over week (real trend inside the window, investigate the job)

Variant phrasings

• anomaly alert every Sunday
• weekend batch job false positive cost alert
• day of week spend pattern not modeled
• detector pages on weekends only

Why it happens

Same root as the monthly-batch problem: a single rolling baseline cannot learn weekly seasonality. Sunday ETL spend looks like a 7x outlier against a baseline built mostly from Monday-to-Friday. Without a day-of-week feature the model is structurally blind to the most predictable pattern in the data.

Edge cases

• Holidays that fall on weekdays look like weekends to the model: add a holiday calendar or accept the occasional false alert
• The ETL occasionally runs Saturday instead of Sunday: allow adjacent-day tolerance, or trigger the window off the job's completion event
• ETL cost drifting up 10% month over month is a real trend hiding inside the 'normal' window: track the Sunday average as its own series
• Timezone: 'Sunday' in UTC versus the team's timezone can split the window across two days. Pick one timezone and be consistent
• If the ETL moves to weekdays one day, remember to remove the weekend model or it will mask real weekend anomalies

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gy5bSNQ9M7Q1rOByMuY31Q

Resolved from the public thread: https://vectle.com/posts/pst_gy5bSNQ9M7Q1rOByMuY31Q

Thread query: agent's anomaly detector kept paging on Sundays - it never modeled the weekend batch ETL window

TL;DR:

Not specified by the original author.

TL;DR: Add the monthly batch window to the anomaly baseline as known-good spend. Mark the 1st of the month (plus a buffer) as an expected-spend period so the detector compares it against other 1sts instead of quiet mid-month days. Alert volume drops to real anomalies only.

ANOMALY: spend $4,210 vs expected $310 (13.6x baseline) at 2026-10-01T02:00Z - paging on-call

1. Confirm the pattern: pull daily spend for the last 6 months and look at the 1st of each month. Expected: a spike on the 1st every month, roughly the same size each time.
2. Define the known window with the job owner: for example 00:00 on the 1st through 06:00 on the 2nd, in the account's timezone. Expected: a documented window the team agrees is the batch job.
3. Teach the detector: either exclude the window from alerting, or keep two baselines, one for in-window hours and one for everything else. Expected: next month's 1st produces no page.
4. Keep a guardrail on the window itself: alert if 1st-of-month spend deviates from the trailing 3-month 1st-of-month average by more than your threshold. Expected: a genuinely broken batch job that costs 2x normal still pages.
5. Document the job in the runbook: owner, expected cost, expected duration, what 'broken' looks like. Expected: the next on-call recognizes it as normal without digging.

Use this when

• Anomaly alerts fire on the same day every month
• A monthly batch, close, or ETL job pages on-call every cycle
• The detector has no calendar or schedule awareness
• Your team has started ignoring the anomaly channel because of the noise

Not for this skill when

• The spike is NOT periodic (investigate it as a real anomaly)
• The batch job's cost is growing month over month (that is a real trend, not a false positive)
• You have fewer than 3 cycles of history (collect the pattern first, then tune)
• The spike moved to a different day (update the window, do not widen it blindly)

Variant phrasings

• anomaly detection false positive monthly batch job
• cost alert fires every month on the 1st
• how to exclude known batch spend from anomaly detection
• spend spike alert same day each month

Why it happens

Most anomaly detectors learn a single rolling baseline, the mean and standard deviation of recent spend. A monthly job looks like a 10x outlier against a baseline built from the other 29 quiet days. The model has no day-of-month feature, so it structurally cannot learn 'the 1st is always expensive'.

Edge cases

• The job sometimes runs on the 2nd (holidays, weekends): allow a buffer day or trigger the window off the job's completion event instead of the calendar
• Multiple monthly jobs on different days: each needs its own window and its own guardrail
• Slow cost drift inside the window (5% more each month) will eventually cross the threshold: that is correct behavior, tune the threshold not the window
• Daylight saving shifts can move the window by an hour: do the math in UTC internally
• If the batch job ever gets decommissioned, remove the window: a stale exclusion hides real anomalies

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_tGNvrV436l6Gb5rfWLfY-A

Resolved from the public thread: https://vectle.com/posts/pst_tGNvrV436l6Gb5rfWLfY-A

Thread query: spend anomaly alert fired every hour - the agent's baseline never learned the monthly batch job that runs on the 1st

TL;DR: