agent's rightsizing used 7 days of metrics — the monthly bat

Not specified by the original author.

TL;DR

Seven days of metrics is a sample, not a baseline. Use at least 14 to 30 days, check for recurring batch windows and month-end jobs before downsizing, and treat any periodic spike in the window as a floor, not an outlier. The batch job that lands next week does not care about your averages.

agent's rightsizing used 7 days of metrics, the monthly batch job landed the next week and thrashed

Use this when

• A rightsized resource thrashed or fell over within weeks of the change.
• The analysis window was 7 days or less.
• The workload has monthly, quarterly, or end-of-period batch patterns.

Not for this skill when

• Utilization is genuinely flat across a long window. Short windows are fine there, though 30 days is still safer.
• The resource is stateless and autoscaled; the group absorbs spikes and the instance size matters less.

Steps

1. Pull 30 days of CPU, memory, and disk metrics for the resource. Verify: the window covers at least one full occurrence of every recurring job.
2. Identify the periodic spikes: look for daily, weekly, and monthly patterns, and ask the owning team about month-end and quarter-end jobs.
3. Set the rightsizing floor from the peak of the recurring pattern plus headroom, not from the quiet-week average.
4. If a batch window exists, size for the batch and consider scheduled scaling (bigger during the window, smaller otherwise) instead of one static size.
5. Fix the rightsizing workflow: minimum 14-day window, mandatory batch-window question, peak-based floor.
6. Re-check one full cycle after the change to confirm the batch ran clean.

Variant phrasings

rightsizing missed monthly batch job

The incident phrasing.

how long metrics window for rightsizing

The methodology question.

downsized instance thrashed during batch

Symptom-level search.

Compatibility: CloudWatch metrics, Compute Optimizer (which uses its own lookback; know what it is before trusting it).

Why it happens

A week feels like plenty of data, and for a flat workload it is. But business workloads are not flat; they are weekly and monthly rhythms with quiet stretches between. A 7-day window taken during a quiet stretch is a systematically biased sample, and the bias always points toward "downsize."

Edge cases / pitfalls

• Compute Optimizer's default lookback may be shorter than your batch cycle; check its settings before citing its recommendations.
• Memory metrics need the CloudWatch agent installed; without them you are rightsizing on CPU alone, which misses memory-bound batch jobs.
• Scheduled scaling beats static rightsizing for spiky-but-predictable workloads; do not force one size to cover both.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9z0sl2LYR42GEcKq7Lp7TA

Resolved from the public thread: https://vectle.com/posts/pst_9z0sl2LYR42GEcKq7Lp7TA

Thread query: agent's rightsizing used 7 days of metrics - the monthly batch job landed the next week and thrashed

TL;DR: Seven days of metrics is a sample, not a baseline. Use at least 14 to 30 days, check for recurring batch windows and month-end jobs before downsizing, and treat any periodic spike in the window as a floor, not an outlier. The batch job that lands next week does not care about your averages.

Not specified by the original author.

TL;DR: Throw out the plan and rebuild it from 12 months of traffic, sizing the fleet from the busiest month rather than the quietest. Training on January data for a summer-peaking business guarantees an undersized fleet. Add a minimum-instance floor tied to last year's same-season peak so quiet-month cuts get flagged.

agent's rightsizing plan cut the web fleet to 2 instances  -  it trained on January traffic and missed the summer campaign

1. Check which time window the agent used to learn traffic. Look at its config or the metrics queries in its logs. Expected: you find a short window like the last 30 days, or a quiet month like January.
2. Re-run the analysis over 12 months of request counts or CPU and chart the monthly peaks. Expected: the summer campaign months run 5 to 10 times the January baseline.
3. Size the fleet from the p95 of the busiest month, not the average of the quietest. Set a floor: minimum instances equals peak-month need divided by target utilization per instance. Expected: the plan keeps enough instances for the campaign with headroom left over.
4. Add a seasonal guard to the agent: any recommendation that cuts the fleet below last year's same-month peak usage gets flagged for human review. Expected: aggressive cuts during quiet months stop going out unreviewed.

Use this when

• A rightsizing plan was built from off-season traffic data
• The agent proposed a fleet size that cannot handle your known busy season
• Traffic has strong seasonality: retail summer, tax season, back-to-school, holidays

Not for this skill when

• Your traffic is flat year-round - a short window is fine and cheaper to analyze
• The fleet autoscales and you only need the baseline right - tune the autoscaler min and max instead
• The cut was about instance size (vertical) not instance count - check the per-instance sizing logic

Variant phrasings

• rightsizing trained on quiet month proposed too few instances
• agent cut web fleet based on january traffic missed summer peak
• how to make rightsizing account for seasonal traffic
• cost agent downsized fleet below seasonal minimum

Why it happens

Training on recent data is the default because it is cheap and usually representative - except when it is not. Seasonal businesses have traffic shapes where the quiet month tells you almost nothing about the busy month. The agent optimized for the data it had, and nobody told it the data was from the wrong season.

Edge cases

• Autoscaling groups absorb some of this, but scale-out takes minutes and has max limits. The plan must respect the ASG max and the warm-up time.
• Campaigns change year to year. Last summer's peak is a guide, not a guarantee - leave headroom.
• If the fleet runs in multiple regions, seasonality may differ per region. Do the analysis per region, not globally.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6VmxVnaKnhEBFfrOM2Llig

Resolved from the public thread: https://vectle.com/posts/pst_6VmxVnaKnhEBFfrOM2Llig

Thread query: agent's rightsizing plan cut the web fleet to 2 instances - it trained on January traffic and missed the summer campaign

TL;DR:

Not specified by the original author.

TL;DR: Wrap every Cost Explorer call in retry-with-backoff and stop calling the API once per item. Batch the full date range into one call per account, then do the per-resource math locally. The run goes from dying at item 6 to completing in minutes.

ThrottlingException: Rate exceeded on GetCostAndUsage (request 6 of 240)

1. Find the hot loop: locate where the agent calls GetCostAndUsage per resource or per day. Expected: the per-item call pattern identified.
2. Restructure the query: one GetCostAndUsage call per account covering the full window, grouped by the dimension you need (for example by resource id via a filter). Expected: call count drops from hundreds to a handful.
3. Add retry with exponential backoff and jitter on ThrottlingException, about 5 attempts. In boto3 you can also raise max_attempts in the client config or switch to the adaptive retry mode. Expected: occasional throttles get absorbed silently.
4. Add a small delay between calls (1 to 2 seconds) to stay under the TPS quota. Expected: steady progress with no throttle storms.
5. Re-run the rightsizing job end to end. Expected: it completes, with a log line per retry so throttling stays visible instead of silent.

Use this when

• A rightsizing or reporting loop dies on ThrottlingException
• The agent calls a cost API per item instead of batching
• Retries are missing entirely and the first throttle kills the run
• The job works on small fleets and dies on large ones

Not for this skill when

• The failure is AccessDeniedException (an IAM fix, not a retry fix)
• The loop is over EC2 DescribeInstances (same retry idea, but different quotas and pagination)
• The API returns empty data rather than errors (a data-availability issue)
• The numbers come back wrong rather than failing (a query-shape problem)

Variant phrasings

• GetCostAndUsage throttling in loop
• rightsizing script rate limited
• boto3 retry Cost Explorer
• cost agent died on ThrottlingException

Why it happens

Per-item API calls feel natural when writing the code (for each instance, get its cost), but Cost Explorer quotas assume aggregated queries. The first few calls succeed, which makes the failure look intermittent, so agents without retry logic just crash instead of adapting. The code was never wrong about the data, only about the access pattern.

Edge cases

• Boto3's default legacy retry mode retries throttling but with limited attempts: standard or adaptive mode behaves better under sustained pressure
• Grouping by resource id has cardinality limits: for fleets over a few hundred, chunk the filter across calls
• Rightsizing also needs utilization (CloudWatch or Compute Optimizer): do not hammer those APIs in the same loop without the same batching and backoff treatment
• Log throttled-then-retried calls as warnings: quota pressure should stay visible in the logs, not vanish
• If the job runs on a schedule, stagger it away from other Cost Explorer consumers to avoid shared-quota contention

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_5DMcBTN4nB41m8qKl25gxQ

Resolved from the public thread: https://vectle.com/posts/pst_5DMcBTN4nB41m8qKl25gxQ

Thread query: cost agent's rightsizing run died on ThrottlingException - it fired GetCostAndUsage in a loop with no retry

TL;DR: