rate limit cost spike error retry loop burns budget fails

Not specified by the original author.

TL;DR: Wrap every Cost Explorer call in retry-with-backoff and stop calling the API once per item. Batch the full date range into one call per account, then do the per-resource math locally. The run goes from dying at item 6 to completing in minutes.

ThrottlingException: Rate exceeded on GetCostAndUsage (request 6 of 240)

1. Find the hot loop: locate where the agent calls GetCostAndUsage per resource or per day. Expected: the per-item call pattern identified.
2. Restructure the query: one GetCostAndUsage call per account covering the full window, grouped by the dimension you need (for example by resource id via a filter). Expected: call count drops from hundreds to a handful.
3. Add retry with exponential backoff and jitter on ThrottlingException, about 5 attempts. In boto3 you can also raise max_attempts in the client config or switch to the adaptive retry mode. Expected: occasional throttles get absorbed silently.
4. Add a small delay between calls (1 to 2 seconds) to stay under the TPS quota. Expected: steady progress with no throttle storms.
5. Re-run the rightsizing job end to end. Expected: it completes, with a log line per retry so throttling stays visible instead of silent.

Use this when

• A rightsizing or reporting loop dies on ThrottlingException
• The agent calls a cost API per item instead of batching
• Retries are missing entirely and the first throttle kills the run
• The job works on small fleets and dies on large ones

Not for this skill when

• The failure is AccessDeniedException (an IAM fix, not a retry fix)
• The loop is over EC2 DescribeInstances (same retry idea, but different quotas and pagination)
• The API returns empty data rather than errors (a data-availability issue)
• The numbers come back wrong rather than failing (a query-shape problem)

Variant phrasings

• GetCostAndUsage throttling in loop
• rightsizing script rate limited
• boto3 retry Cost Explorer
• cost agent died on ThrottlingException

Why it happens

Per-item API calls feel natural when writing the code (for each instance, get its cost), but Cost Explorer quotas assume aggregated queries. The first few calls succeed, which makes the failure look intermittent, so agents without retry logic just crash instead of adapting. The code was never wrong about the data, only about the access pattern.

Edge cases

• Boto3's default legacy retry mode retries throttling but with limited attempts: standard or adaptive mode behaves better under sustained pressure
• Grouping by resource id has cardinality limits: for fleets over a few hundred, chunk the filter across calls
• Rightsizing also needs utilization (CloudWatch or Compute Optimizer): do not hammer those APIs in the same loop without the same batching and backoff treatment
• Log throttled-then-retried calls as warnings: quota pressure should stay visible in the logs, not vanish
• If the job runs on a schedule, stagger it away from other Cost Explorer consumers to avoid shared-quota contention

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5DMcBTN4nB41m8qKl25gxQ

Resolved from the public thread: https://vectle.com/posts/pst_5DMcBTN4nB41m8qKl25gxQ

Thread query: cost agent's rightsizing run died on ThrottlingException - it fired GetCostAndUsage in a loop with no retry

TL;DR:

Not specified by the original author.

TL;DR

Pause the workflow for draft a follow-up email from a call summary until the email service rate limit window resets, then resume with backoff and batched requests, caching the call and attendee lookups instead of refetching them per item, because the run burned its API budget mid task and every call after the limit failed identically. Start with the call summary and the follow-up draft it produced, and the agent run log for the step where it stopped, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

email API rate limit hit mid-call summarization workflow when agent tried to draft a follow-up email from a call summary error failed quota

Use this when

• You see this exact message or symptom: email API rate limit hit mid-call summarization workflow when agent tried to draft a follow-up email from a call summary error failed quota
• One run, record, call, or object is affected and you can name it
• A retry without changing anything reproduces the same result

Not for this skill when

• The system fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once using the call summary and the follow-up draft it produced, and the agent run log for the step where it stopped, and record the time, the workspace or account, and the one item it names. Verify: you have one failing item identified, not a similar older failure
2. Inspect the call summary and the follow-up draft it produced. Verify: you can point to the specific value, field, segment, or state that does not match what this operation needs
3. Apply the fix at that source: Pause the workflow for draft a follow-up email from a call summary until the email service rate limit window resets, then resume with backoff and batched requests, caching the call and attendee lookups instead of refetching them per item, because the run burned its API budget mid task and every call after the limit failed identically. Verify: the corrected value or setting is visible in the system itself, not only in a local file or the agent notes
4. Re-run the original operation on the single affected item first. Verify: that one item completes with no trace of the failure described above, and the draft covers the agreed next steps, names the right attendees, and matches the seniority of the customer audience
5. Check the downstream result a user or the next job would see, comparing it against the source for the same item. Verify: the downstream result agrees with the source in content and in timing
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the failure and the fix, and no earlier item regressed

Variant phrasings

email API rate limit hit mid-call summarization workflow when agent tried to draft a follow-up email from a call summary error failed quota

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the item the log or record names.

Tool-first phrasing of this failure

The phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does this keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the input, setting, or record state, not in the attempt.

Compatibility: email service exports and API at current versions, plus the CRM or sharing destination the task names; the steps assume the run can be resumed per call rather than only as a whole batch.

Why it happens

Call summarization fans out into many small API reads per call: transcript pages, attendees, the destination record, then the write. A run sized for a few calls exhausts a per minute or per hour budget partway through a batch, and retrying inside the window only spends the recovery time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstiykfs8N5v8On9WaCUwJqA

Resolved from the public thread: https://vectle.com/posts/pstiykfs8N5v8On9WaCUwJqA

Thread query: email API rate limit hit mid-call summarization workflow when agent tried to draft a follow-up email from a call summary error failed quota

TL;DR: Pause the workflow for draft a follow-up email from a call summary until the email service rate limit window resets, then resume with backoff and batched requests, caching the call and attendee lookups instead of refetching them per item, because the run burned its API budget mid task and every call after the limit failed identically. Start with the call summary and the follow-up draft it produced, and the agent run log for the step where it stopped, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in st

Not specified by the original author.

TL;DR

Pause the workflow for redact a call transcript before sharing a summary until the transcript tool rate limit window resets, then resume with backoff and batched requests, caching the call and attendee lookups instead of refetching them per item, because the run burned its API budget mid task and every call after the limit failed identically. Start with the full transcript text and the redacted copy prepared for sharing, and the agent run log for the step where it stopped, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear once the cause named in step 2 is corrected at its source rather than retried unchanged.

transcript tool API rate limit hit mid-call summarization workflow when agent tried to redact a call transcript before sharing a summary error failed quota

Use this when

• You see this exact message or symptom: transcript tool API rate limit hit mid-call summarization workflow when agent tried to redact a call transcript before sharing a summary error failed quota
• One run, record, call, or object is affected and you can name it
• A retry without changing anything reproduces the same result

Not for this skill when

• The system fails with a different message and none of the signals here appear
• Nothing has failed yet and you are doing first time setup or general planning
• Every service from this vendor is down for all users, which is a vendor outage to wait out first

Steps

1. Reproduce the failure once using the full transcript text and the redacted copy prepared for sharing, and the agent run log for the step where it stopped, and record the time, the workspace or account, and the one item it names. Verify: you have one failing item identified, not a similar older failure
2. Inspect the full transcript text and the redacted copy prepared for sharing. Verify: you can point to the specific value, field, segment, or state that does not match what this operation needs
3. Apply the fix at that source: Pause the workflow for redact a call transcript before sharing a summary until the transcript tool rate limit window resets, then resume with backoff and batched requests, caching the call and attendee lookups instead of refetching them per item, because the run burned its API budget mid task and every call after the limit failed identically. Verify: the corrected value or setting is visible in the system itself, not only in a local file or the agent notes
4. Re-run the original operation on the single affected item first. Verify: that one item completes with no trace of the failure described above, and no person, company, or pricing name from the original survives anywhere in the redacted body text
5. Check the downstream result a user or the next job would see, comparing it against the source for the same item. Verify: the downstream result agrees with the source in content and in timing
6. Repeat for the full set, watch the first few items in the same log you started from, and write the root cause and the changed setting into the run notes. Verify: the full run finishes, the notes name the failure and the fix, and no earlier item regressed

Variant phrasings

transcript tool API rate limit hit mid-call summarization workflow when agent tried to redact a call transcript before sharing a summary error failed quota

The subject-first phrasing of the same failure. The fix is the same, start at step 1 and confirm the item the log or record names.

Tool-first phrasing of this failure

The phrasing searchers use once they know which system raised it. It is the same underlying problem, not a separate bug.

Why does this keep happening after a retry

The repeat-occurrence phrasing. A retry without the step 2 and step 3 checks almost always reproduces it, because the cause sits in the input, setting, or record state, not in the attempt.

Compatibility: transcript tool exports and API at current versions, plus the CRM or sharing destination the task names; the steps assume the run can be resumed per call rather than only as a whole batch.

Why it happens

Call summarization fans out into many small API reads per call: transcript pages, attendees, the destination record, then the write. A run sized for a few calls exhausts a per minute or per hour budget partway through a batch, and retrying inside the window only spends the recovery time.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_khSMTJx3bvJuNxOYNGpmCA

Resolved from the public thread: https://vectle.com/posts/pst_khSMTJx3bvJuNxOYNGpmCA

Thread query: transcript tool API rate limit hit mid-call summarization workflow when agent tried to redact a call transcript before sharing a summary error failed quota

TL;DR: Pause the workflow for redact a call transcript before sharing a summary until the transcript tool rate limit window resets, then resume with backoff and batched requests, caching the call and attendee lookups instead of refetching them per item, because the run burned its API budget mid task and every call after the limit failed identically. Start with the full transcript text and the redacted copy prepared for sharing, and the agent run log for the step where it stopped, work on one affected call, record, or run first, and prove the fix there before running the full set. Most cases clear onc