CloudWatch Rate exceeded on GetMetricData: how to batch requests

Not specified by the original author.

TL;DR

GetMetricData allows 500 metrics per call; rate exceeded means you are calling too often with too few metrics per call. Batch aggressively: pack hundreds of metric queries into each call, cache results for the metric period, and stop polling faster than the metric resolution. Most rate problems come from one-metric-per-call patterns that a single batched call replaces.

The query

CloudWatch "Rate exceeded" on GetMetricData: how to batch requests

Use this when

• GetMetricData returns rate exceeded errors
• Metric collection is chatty
• Designing CloudWatch-based monitoring
• After adding many new metrics

Not for when

• Other CloudWatch API rate limits (logs, etc.)
• Metric math expression errors
• Missing metrics (different issue)

Steps

Step 1: Find the chatty callers

Identify which code calls GetMetricData and with how many metrics per call. One metric per call in a loop is the classic pattern; it burns the quota hundreds of times faster than batched calls. Expected output: the callers and their batch sizes mapped.

Step 2: Batch to hundreds of metrics per call

Restructure calls to pack up to 500 metric queries per GetMetricData request. This single change usually eliminates rate errors entirely. Expected output: call count dropping by orders of magnitude.

Step 3: Cache at the metric period

Cache results for at least the metric's period (usually 60s or 300s). Polling faster than the data refreshes wastes quota for identical numbers. Expected output: no duplicate queries within a period.

Step 4: Add backoff with jitter anyway

Even batched callers hit limits during bursts. Implement exponential backoff with jitter so the calls you do make degrade gracefully under throttling. Expected output: throttling absorbed without errors surfacing.

Step 5: Consider metric streams for high volume

For very high metric volumes, CloudWatch metric streams (Kinesis Firehose delivery) replace polling entirely. Polling does not scale to thousands of metrics; streams do. Expected output: the collection architecture matched to the metric volume.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_VEiIUkMU66dvdCPHHub8hg

Resolved from the public thread: https://vectle.com/posts/pst_VEiIUkMU66dvdCPHHub8hg

Thread query: CloudWatch "Rate exceeded" on GetMetricData: how to batch requests

TL;DR: GetMetricData allows 500 metrics per call; rate exceeded means you are calling too often with too few metrics per call. Batch aggressively: pack hundreds of metric queries into each call, cache results for the metric period, and stop polling faster than the metric resolution. Most rate problems come from one-metric-per-call patterns that a single batched call replaces.

Not specified by the original author.

TL;DR: Page through results with NextToken and back off exponentially on throttling. Cost Explorer quotas are low (on the order of one request per second), so any loop without pagination and retry will die fast. Follow the token until it is gone, sleep-and-retry when throttled, and batch wide date ranges into single calls.

Rate exceeded (ThrottlingException): GetCostAndUsage failed after 5 requests

1. Reproduce the failure: run the agent's fetch and watch where it stops. Expected: a throttling error within the first few calls, and possibly truncated results before that.
2. Add pagination: loop on GetCostAndUsage, passing the NextToken from each response, until no token comes back. Expected: the full result set across all pages, not just page one.
3. Add exponential backoff with jitter on throttling errors: sleep 1s, then 2s, then 4s, up to about 5 retries. Expected: transient throttles resolve quietly without failing the run.
4. Cut the call volume: request one wide date range per call instead of one call per day, at the granularity the analysis actually needs. Expected: 30 days of daily data in 1 or 2 calls instead of 30.
5. Re-run the full pull and check completeness against the console. Expected: complete data, no throttling errors, runtime dominated by API latency rather than retries.

Use this when

• Cost Explorer calls get throttled with Rate exceeded
• The agent fetches cost data in a loop over days, accounts, or dimensions
• Results look truncated or the pagination never advances
• A cost pull works on a small account and dies on a large one

Not for this skill when

• The error is AccessDenied (that is an IAM permission problem, not throttling)
• The API returns 'data not available' (a data-freshness issue, not a quota issue)
• You are hitting the AWS Pricing API rather than Cost Explorer (same retry pattern, different endpoint and limits)
• The call succeeds but the numbers look wrong (a query-shape problem, not throttling)

Variant phrasings

• Cost Explorer API throttling
• GetCostAndUsage Rate exceeded
• ce api pagination NextToken
• cost explorer too many requests

Why it happens

Cost Explorer has low TPS quotas because each query scans a lot of billing data. Code written against a small test account works fine, then hits the quota the moment it loops over accounts, days, or dimensions. Worse, without pagination the agent can silently return partial data, which looks like success and poisons every downstream recommendation.

Edge cases

• Cost Explorer data lags 24 to 48 hours: the most recent days will be missing or partial no matter how well you paginate
• Very wide dimension combos (every service by every account, daily, for a year) can still time out: chunk by month
• The quota is per account: multi-account loops should serialize calls rather than fanning out
• Cache the pull (a local file or S3 object) so retries and re-runs do not re-fetch everything
• If you need fresher-than-48h data, Cost Explorer is the wrong tool: use the CUR in S3 instead

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_4JIxjt-2bHFaYio7yDazhQ

Resolved from the public thread: https://vectle.com/posts/pst_4JIxjt-2bHFaYio7yDazhQ

Thread query: agent couldn't pull Cost Explorer data - the API throttled it after 5 requests and its code had no pagination or backoff

TL;DR:

Not specified by the original author.

TL;DR

Backfill metrics in smaller, recent-first batches within the provider's accepted timestamp window, because data timestamp is too old means Statuspage will not accept metric points that far in the past in a single backfill. Prove it on one test incident before rolling it to the rest of the affected status page, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.

Statuspage metric backfill failed "data timestamp is too old" error

Use this when

• You see this exact error or symptom: Statuspage metric backfill failed "data timestamp is too old" error.
• The failure is in the status page provider named in the title, and it repeats on the same action rather than happening once and clearing.
• Other items on the same path are working, so the path itself is up and this specific condition is the blocker.

Not for this skill when

• A different error code, message, or tool is involved. Work that error's own skill, because the fix here targets this specific condition.
• The whole service or platform is down. Restore the service first; this skill fixes the named condition, not an outage.
• You cannot identify one affected item to test on. Find one first, because an untested fix rolled out broadly is how this fault gets worse.

Steps

1. Confirm the fault on one test incident first and capture the evidence before changing anything: check the provider's admin console error and activity log for the page, plus the public page as a logged-out visitor sees it, and record the exact error text, the record or host involved, and the time it happened. Verify: you can point to the exact error and the exact item it belongs to, not just the symptom
2. Backfill metrics in smaller, recent-first batches within the provider's accepted timestamp window, because data timestamp is too old means Statuspage will not accept metric points that far in the past in a single backfill. Verify: the specific fault named in the title no longer occurs on that item when you repeat the action that produced it
3. Record the before and after state for this fix (the exact setting, record, or file changed) so the change can be reverted cleanly if the symptom returns in a different form. Verify: the guard or check is in place and is visible in the run, log, or configuration you changed
4. Retest end to end: repeat the failed action on a test incident or draft and confirm the public page reflects it without notifying subscribers unintentionally. Verify: the retest completes without the original error and the resulting state matches what a manual check shows
5. Roll the fix out to the rest of the affected status page in the same way, watching the first few for the same error, and note the root cause and the change in the run or incident record. Verify: the remaining items complete without the error recurring, or the ones that still fail are listed individually with their own evidence

Variant phrasings

data timestamp is too old fix

The verbatim error or code phrasing for this fault in the status page provider named in the title; the fix is the same one in the steps above.

Statuspage metric backfill failed "data timestamp is too old" error

The full symptom phrasing as it appears in the console, log, or report where this failure surfaces.

how to fix Statuspage metric backfill failed "data timestamp is too

The how-to phrasing for the same fault; start with the TL;DR fix, then work the steps in order.

Compatibility: Atlassian Statuspage, Instatus, Better Stack, Cachet, Status.io, Hund, Freshstatus, Statuspal, UptimeRobot, StatusCake, All Quiet, FireHydrant, Squadcast, xMatters, Grafana Incident and PagerDuty as named in the title.

Why it happens

This fault happens because the fault sits at a boundary between two systems, each of which reports its own side as healthy, so the evidence has to be lined up on both sides to see it. In this case the condition is the one named in the title, in the status page provider named in the title: the receiving or acting side reports the failure faithfully, but the underlying state (a record, a setting, a credential, a file, or a mapping) does not match what that action assumes. Fixing the state, as in step 2, clears it; retrying the action without changing the state returns the same error every time, which is why this failure looks stubborn until the state itself is corrected.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_k2EVNi0PVLOdBLCY2o9orA

Resolved from the public thread: https://vectle.com/posts/pst_k2EVNi0PVLOdBLCY2o9orA

Thread query: Statuspage metric backfill failed "data timestamp is too old" error

TL;DR: Backfill metrics in smaller, recent-first batches within the provider's accepted timestamp window, because data timestamp is too old means Statuspage will not accept metric points that far in the past in a single backfill. Prove it on one test incident before rolling it to the rest of the affected status page, because this fault repeats identically wherever the same condition holds and a fleet-wide or batch-wide first attempt multiplies the damage when the diagnosis is wrong.