agent's cost export to s3 failed silently - the finops dashboard showed last month's data for a week
Fixes a silently failing cost export to S3 by adding failure alerting and data-freshness monitoring. Use when a cost export to S3 fails silently and dashboards go stale. Not for exports that fail loudly.
TL;DR
The export job failed without notifying anyone, so the dashboard kept rendering last month's data for a week before anyone noticed. Fix the export failure itself, then add two monitors: alert on export job failure, and alert when the dashboard's latest data date falls behind schedule. Silent failures need loud monitors.
Error
agent's cost export to s3 failed silently - the finops dashboard showed last month's data for a weekSteps
- Find the export failure: check the export job's run history for the first failed run. Expected: the failure and its cause are identified.
- Fix the underlying cause (credentials, permissions, destination config). Expected: the export runs successfully.
- Backfill the missing period so the dashboard is current. Expected: the data gap is closed.
- Add alerting on export job failure (nonzero exit, error status) to the on-call channel. Expected: the next failure pages someone.
- Add a data-freshness check: alert if the newest data in the dashboard is older than expected. Expected: staleness is caught even when the job reports success.
When to use
- A cost export fails without alerting.
- Dashboards show stale data unnoticed.
- The export job lacks failure notification.
When not to use
- The export fails loudly already (fix the cause, not the alerting).
- The dashboard queries the wrong location (a configuration problem).
- The data is current (no problem).
Tool compatibility
AWS Cost and Usage Reports, S3, dashboard tools (current 2026 releases).
Variant phrasings
export failed and nobody knew
Alert on job failure.
dashboard stale for a week
Add a data-freshness monitor.
silent pipeline failure
Monitor the output's freshness, not just the job's status.
Why it happens
Export jobs fail for mundane reasons (rotated credentials, full quotas, config drift), and without failure alerting the failure is invisible: the dashboard happily renders old data because old data is still valid data. Freshness monitoring catches the class (stale output) even when job-status monitoring misses the instance (a job that "succeeds" while writing nothing). Both monitors are cheap; the week of blind decisions is expensive.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_GKj43Zzl0EWMKcCIipzTSQ