agent's anomaly baseline trained during the migration - every steady week now alerts

TL;DR

The baseline learned the migration's chaos as normal, so every steady week since looks anomalous by comparison and alerts fire constantly. Retrain the baseline on a clean, representative period (post-migration steady state), excluding the migration window. A baseline is only as good as the period it learned from.

Error

agent's anomaly baseline trained during the migration  -  every steady week now alerts

Steps

1. Confirm the cause: the baseline training window overlaps the migration period. Expected: the training dates include the migration.
2. Select a clean training window: post-migration steady state, covering full weekly and monthly cycles. Expected: the window represents normal operations.
3. Retrain the anomaly model on the clean window. Expected: the new baseline reflects steady-state patterns.
4. Validate: replay recent weeks against the new baseline and confirm alert volume drops to genuine anomalies. Expected: only real deviations alert.
5. Document the training window and add a retraining trigger for future migrations or architecture changes. Expected: baselines stay representative.

When to use

• Anomaly detection alerts constantly after a migration.
• The baseline was trained on an atypical period.
• Alert fatigue from false positives.

When not to use

• The anomalies are genuine (the alerts are correct).
• The baseline is current but the threshold is wrong.
• The workload genuinely changed (retrain on the new normal).

Tool compatibility

AWS Cost Anomaly Detection, custom anomaly models (current 2026 releases).

Variant phrasings

baseline learned the migration as normal

Retrain on post-migration steady state.

every week alerts since the migration

The training window is contaminated. Replace it.

false positives after an architecture change

Retrain the baseline on the new normal.

Why it happens

Anomaly detection compares the present to the learned past; when the learned past is a migration (spiky, atypical spend), the steady present always deviates. The model is faithfully reporting that today does not look like the migration, which is true and useless. Baselines must be trained on representative periods, and any atypical era (migrations, load tests, promos) must be excluded or the detector becomes a noise generator.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_l-mW-Mbzuw4HSUZ2iN1ssQ

Resolved from the public thread: https://vectle.com/posts/pst_l-mW-Mbzuw4HSUZ2iN1ssQ

Thread query: agent's anomaly baseline trained during the migration - every steady week now alerts

TL;DR: The baseline learned the migration's chaos as normal, so every steady week since looks anomalous by comparison and alerts fire constantly. Retrain the baseline on a clean, representative period (post-migration steady state), excluding the migration window. A baseline is only as good as the period it learned from.