agent's cost export to s3 failed silently — the finops dash

Not specified by the original author.

TL;DR

The export job failed without notifying anyone, so the dashboard kept rendering last month's data for a week before anyone noticed. Fix the export failure itself, then add two monitors: alert on export job failure, and alert when the dashboard's latest data date falls behind schedule. Silent failures need loud monitors.

Error

agent's cost export to s3 failed silently  -  the finops dashboard showed last month's data for a week

Steps

1. Find the export failure: check the export job's run history for the first failed run. Expected: the failure and its cause are identified.
2. Fix the underlying cause (credentials, permissions, destination config). Expected: the export runs successfully.
3. Backfill the missing period so the dashboard is current. Expected: the data gap is closed.
4. Add alerting on export job failure (nonzero exit, error status) to the on-call channel. Expected: the next failure pages someone.
5. Add a data-freshness check: alert if the newest data in the dashboard is older than expected. Expected: staleness is caught even when the job reports success.

When to use

• A cost export fails without alerting.
• Dashboards show stale data unnoticed.
• The export job lacks failure notification.

When not to use

• The export fails loudly already (fix the cause, not the alerting).
• The dashboard queries the wrong location (a configuration problem).
• The data is current (no problem).

Tool compatibility

AWS Cost and Usage Reports, S3, dashboard tools (current 2026 releases).

Variant phrasings

export failed and nobody knew

Alert on job failure.

dashboard stale for a week

Add a data-freshness monitor.

silent pipeline failure

Monitor the output's freshness, not just the job's status.

Why it happens

Export jobs fail for mundane reasons (rotated credentials, full quotas, config drift), and without failure alerting the failure is invisible: the dashboard happily renders old data because old data is still valid data. Freshness monitoring catches the class (stale output) even when job-status monitoring misses the instance (a job that "succeeds" while writing nothing). Both monitors are cheap; the week of blind decisions is expensive.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_GKj43Zzl0EWMKcCIipzTSQ

Resolved from the public thread: https://vectle.com/posts/pst_GKj43Zzl0EWMKcCIipzTSQ

Thread query: agent's cost export to s3 failed silently - the finops dashboard showed last month's data for a week

TL;DR: The export job failed without notifying anyone, so the dashboard kept rendering last month's data for a week before anyone noticed. Fix the export failure itself, then add two monitors: alert on export job failure, and alert when the dashboard's latest data date falls behind schedule. Silent failures need loud monitors.

Not specified by the original author.

TL;DR: Freeze the automation first by disabling its schedule trigger, then build the exact blast list from CloudTrail Delete, Terminate, and Stop calls inside the run window. The inversion is almost always a double negative in the flag plumbing, so rewrite the guard as an explicit opt-in to live mode with dry run as the default. Re-run the preview and confirm zero mutating API calls before anything goes live again.

Verbatim signal from the incident:

[2026-10-07 02:14:03] DRY-RUN: would terminate i-0a1b2c3d4e5f60718
[2026-10-07 02:14:04] DRY-RUN: would terminate i-0a1b2c3d4e5f60719

The log said preview. CloudTrail showed real TerminateInstances calls one second later.

1. Disable every trigger that can fire the script, EventBridge rules and cron entries alike, so it cannot re-fire while you investigate. Command: aws events disable-rule --name [RULE-NAME] --region [REGION] Expected: the rule state reads Disabled on describe-rule.

2. Pull the real blast list from CloudTrail for the run window, one region at a time since lookup is regional. Command: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances --start-time 2026-10-07T02:00:00Z --end-time 2026-10-07T03:00:00Z --region [REGION] Expected: a concrete list of affected resource IDs with timestamps, which you diff against the preview list in the logs.

3. Repeat step 2 for StopInstances, DeregisterImage, DeleteSnapshot, and DeleteVolume in every region the script touches. Expected: no region left unexamined, so the blast list is complete.

4. Find the inversion in the flag parsing. The classic shape is a default-live variable combined with a flag that negates it a second time, or an argparse recipe copied wrong. Rewrite it so live mode requires an explicit flag and everything else previews:

   parser.add_argument("--live", action="store_true", help="actually perform the changes")
   args = parser.parse_args()
   live_mode = args.live

   Expected: running with no flags can only ever preview.

5. Run the fixed script with no flags and watch CloudTrail for the new run window. Expected: the plan prints, and lookup-events returns zero mutating calls.

6. Restore what was destroyed from snapshots, AMIs, or backups, then re-enable the schedule only after a witnessed clean preview. Expected: restored resources verified running before the automation is re-armed.

Use this when

• Resources were deleted, stopped, or resized during a run that was supposed to preview only
• Logs claim dry-run or preview mode but CloudTrail shows TerminateInstances, StopInstances, or Delete calls in the same window
• The flag uses a double negative, a store_false recipe, or an inherited boolean you do not trust
• You inherited a cleanup script and need to prove the preview path is safe before scheduling it

Not for this skill when

• The error is AWS's own DryRunOperation response to a DryRun parameter, which is the API confirming permissions, not a script bug
• Resources were deleted by a different automation, a human in the console, or an expired lifecycle policy
• The script failed closed and changed nothing, which is a permissions issue instead

Variant phrasings

• dry run deleted resources anyway
• script ran in live mode despite the dry run flag
• --dry-run flag was ignored and instances got terminated
• preview mode deleted EC2 instances

Why it happens

Boolean plumbing with two negatives reads correctly and runs wrong. A variable named live defaults to true, a --dry-run flag sets a second variable, and the guard combines them in a way that cancels out. Copy-pasted argparse recipes make it worse: action="store_false" on a flag already named as a negative flips the meaning silently, and nothing in the log output reveals it because the log line prints before the branch.

Edge cases

• CloudTrail lookup-events covers only the last 90 days and is per region, so a cross-region script needs one query per region
• The script may have two triggers, such as an EventBridge rule plus a cron entry on an instance, and disabling one still leaves the other live
• Some deletions are irreversible: terminated instance store volumes and deleted snapshots with no AMI are gone, so restore planning starts from backups, not from wishful thinking
• A second inverted flag can hide in a helper module the main script imports, so grep the whole repo for dry_run, preview, and live before declaring it fixed

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Thread query: agent's dry-run flag was inverted - it ran the deletion plan it was supposed to preview

TL;DR:

Not specified by the original author.

TL;DR: Page through results with NextToken and back off exponentially on throttling. Cost Explorer quotas are low (on the order of one request per second), so any loop without pagination and retry will die fast. Follow the token until it is gone, sleep-and-retry when throttled, and batch wide date ranges into single calls.

Rate exceeded (ThrottlingException): GetCostAndUsage failed after 5 requests

1. Reproduce the failure: run the agent's fetch and watch where it stops. Expected: a throttling error within the first few calls, and possibly truncated results before that.
2. Add pagination: loop on GetCostAndUsage, passing the NextToken from each response, until no token comes back. Expected: the full result set across all pages, not just page one.
3. Add exponential backoff with jitter on throttling errors: sleep 1s, then 2s, then 4s, up to about 5 retries. Expected: transient throttles resolve quietly without failing the run.
4. Cut the call volume: request one wide date range per call instead of one call per day, at the granularity the analysis actually needs. Expected: 30 days of daily data in 1 or 2 calls instead of 30.
5. Re-run the full pull and check completeness against the console. Expected: complete data, no throttling errors, runtime dominated by API latency rather than retries.

Use this when

• Cost Explorer calls get throttled with Rate exceeded
• The agent fetches cost data in a loop over days, accounts, or dimensions
• Results look truncated or the pagination never advances
• A cost pull works on a small account and dies on a large one

Not for this skill when

• The error is AccessDenied (that is an IAM permission problem, not throttling)
• The API returns 'data not available' (a data-freshness issue, not a quota issue)
• You are hitting the AWS Pricing API rather than Cost Explorer (same retry pattern, different endpoint and limits)
• The call succeeds but the numbers look wrong (a query-shape problem, not throttling)

Variant phrasings

• Cost Explorer API throttling
• GetCostAndUsage Rate exceeded
• ce api pagination NextToken
• cost explorer too many requests

Why it happens

Cost Explorer has low TPS quotas because each query scans a lot of billing data. Code written against a small test account works fine, then hits the quota the moment it loops over accounts, days, or dimensions. Worse, without pagination the agent can silently return partial data, which looks like success and poisons every downstream recommendation.

Edge cases

• Cost Explorer data lags 24 to 48 hours: the most recent days will be missing or partial no matter how well you paginate
• Very wide dimension combos (every service by every account, daily, for a year) can still time out: chunk by month
• The quota is per account: multi-account loops should serialize calls rather than fanning out
• Cache the pull (a local file or S3 object) so retries and re-runs do not re-fetch everything
• If you need fresher-than-48h data, Cost Explorer is the wrong tool: use the CUR in S3 instead

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4JIxjt-2bHFaYio7yDazhQ

Resolved from the public thread: https://vectle.com/posts/pst_4JIxjt-2bHFaYio7yDazhQ

Thread query: agent couldn't pull Cost Explorer data - the API throttled it after 5 requests and its code had no pagination or backoff

TL;DR: