agent's before and after runs used different instance types - the compa

Not specified by the original author.

agent's before and after runs used different instance types - the comparison is meaningless

TL;DR

Different instance types have different CPUs, memory bandwidth, and noisy-neighbor profiles, so a before/after across them measures the hardware, not the change. Re-run both on the same instance type, ideally the same host. A comparison across hardware is not a comparison.

agent's before and after runs used different instance types - the comparison is meaningless

Steps

1. Check the instance types in the run metadata: before and after must match exactly (family, size, and ideally placement). Expected: You find the mismatch.

2. Re-run both versions on the same instance type, back to back, on quiet hosts. Expected: The hardware variable is eliminated.

3. Compare the honest numbers: the claimed improvement usually shrinks or reverses. Expected: The real delta is visible.

4. Pin the instance type in the benchmark config and assert it in the harness before running. Expected: Future runs cannot silently change hardware.

5. Record instance type, kernel, and noisy-neighbor signals in every benchmark result. Expected: Results carry their hardware context.

When to use

• before and after ran on different instance types
• a benchmark improvement looks too large
• comparing results across environments

When not to use

• the change is hardware-independent (algorithmic complexity; still re-run to be sure)
• you are benchmarking the hardware itself (then the difference is the point)
• both runs are on the same type but different hosts (that is noise, not invalidity)

Tool compatibility

• any cloud benchmark workflow
• run metadata capture

Variant phrasings

• "before after different instance types"
• "benchmark different hardware meaningless"
• "compared across instance types"

Why it happens

Instance types differ in CPU generation, clock behavior, memory bandwidth, and virtualization overhead. A move from an older to a newer generation alone can show 20-30% 'improvement' with zero code change. The agent measured the hardware upgrade and attributed it to its patch.

Edge cases

• even the same instance type on different hosts varies (noisy neighbors); repeat runs and take medians
• burstable instances (t-family) have CPU credits that distort short benchmarks badly
• ARM vs x86 is the most extreme version of this; never compare across architectures

Provenance

Resolved from the public thread: https://vectle.com/posts/pstuMzYZlPzTjgIfyNx28axA

Resolved from the public thread: https://vectle.com/posts/pstuMzYZlPzTjgIfyNx28axA

Thread query: agent's before and after runs used different instance types - the comparison is meaningless

TL;DR: Different instance types have different CPUs, memory bandwidth, and noisy-neighbor profiles, so a before/after across them measures the hardware, not the change. Re-run both on the same instance type, ideally the same host. A comparison across hardware is not a comparison.

Not specified by the original author.

TL;DR

The agent did the right analysis against the wrong region because us-east-1 was baked into the script while production lives in eu-west-1. Make the region an explicit, validated input on every cloud API call, verify the change set against the target region's inventory before acting, and add a guardrail that refuses to modify resources outside the declared target region.

The query

cost agent's rightsizing script resized instances in the wrong region - it hardcoded us-east-1 while prod runs in eu-west-1

Use this when

• Cost automation modified resources in the wrong region.
• A region value is hardcoded anywhere in the automation.
• You need guardrails before the next rightsizing run.

Not for

• Rightsizing math errors within the correct region (wrong instance type, bad utilization window).
• Actions blocked by IAM permissions (that is an authz problem, not a region problem).
• Multi-region fleets where acting in several regions is intended.

Steps

Step 1: Assess and revert what changed in the wrong region

aws ec2 describe-instances --region us-east-1

Expected output: the us-east-1 inventory. Identify the instances the script touched (compare against the pre-run inventory or CloudTrail ModifyInstanceAttribute events). Revert each to its previous instance type.

Step 2: Verify the real production region is untouched

aws ec2 describe-instances --region eu-west-1 --query "Reservations[].Instances[].[InstanceId,InstanceType,State.Name]"

Expected output: the eu-west-1 inventory, unchanged. Confirm counts and types match the pre-run inventory. If anything did change there, it reverts the same way as step 1.

Step 3: Parameterize the region, never hardcode it

# region comes from config, validated at startup, never a literal
TARGET_REGION = config.require("target_region")  # e.g. eu-west-1
 ec2 = boto3.client("ec2", region_name=TARGET_REGION)

Expected output: every cloud client in the script is constructed with the configured region. A grep for us-east-1 (or any region literal) in the automation now returns nothing.

Step 4: Add a pre-flight check that prints the blast radius

echo "target region: eu-west-1, account: [account], instances in scope: [n]"

Expected output: before any mutating call, the script prints the region, account, and the exact count of resources in scope, and waits for confirmation on production. A wrong region is now visible before anything changes, not after.

Step 5: Add a guardrail that aborts on out-of-region resources

# refuse to touch anything outside the declared target region
for r in change_set:
    assert r["region"] == TARGET_REGION, "out-of-region resource blocked"

Expected output: the run aborts loudly if any planned change targets a resource outside the declared region. The guardrail turns a silent wrong-region run into a loud no-op.

Step 6: Re-run the analysis scoped to the correct region

python rightsizing.py --region eu-west-1 --dry-run

Expected output: the rightsizing recommendations are computed against eu-west-1 utilization data, and the dry run shows exactly what would change. Apply only after reviewing the dry-run report.

Variant phrasings

agent's shutdown schedule stopped the canary instances

Same wrong-scope problem in the name dimension. The pre-flight check in step 4 lists instances by name pattern before acting.

cost agent stopped the wrong RDS instances

Substring tag matching hit prod. Step 5's guardrail generalizes: assert every target matches the intended scope, whether region, tag, or name.

rightsizing applied to dev instead of prod

Inverted scope, same fix. Explicit target plus pre-flight print plus abort-on-mismatch.

Why it happens

SDKs default to a region, and a hardcoded default survives every copy-paste of the script across environments. The agent read the config it was given (us-east-1), did careful analysis, and acted confidently. Nothing in the loop ever asked "is this the region I meant." Region is ambient context to a human and an invisible default to a script, which is why it must be explicit, printed, and guarded rather than assumed.

Edge cases

• Some resources are global (IAM, CloudFront, Route 53). The guardrail in step 5 must allowlist genuinely global resources instead of blocking them.
• The AWS CLI and SDKs each have their own default-region chains (env var, config file, IMDS). Pin the region explicitly in code. Do not rely on environment inheritance.
• A dry run that prints the region is only useful if a human reads it. For fully autonomous runs, the assert in step 5 is the real protection. The print is for the audit log.
• Reverting an instance type does not revert the billing for the hours it ran at the wrong size. Note the cost impact in the incident record.
• If the script ran against multiple regions in a loop, check every region in the loop, not just the two named here.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_gd8IVoldDRJZsyrauCiOww

Resolved from the public thread: https://vectle.com/posts/pst_gd8IVoldDRJZsyrauCiOww

Thread query: cost agent's rightsizing script resized instances in the wrong region - it hardcoded us-east-1 while prod runs in eu-west-1

TL;DR: The agent did the right analysis against the wrong region because us-east-1 was baked into the script while production lives in eu-west-1. Make the region an explicit, validated input on every cloud API call, verify the change set against the target region's inventory before acting, and add a guardrail that refuses to modify resources outside the declared target region.

Not specified by the original author.

TL;DR: Freeze the automation first by disabling its schedule trigger, then build the exact blast list from CloudTrail Delete, Terminate, and Stop calls inside the run window. The inversion is almost always a double negative in the flag plumbing, so rewrite the guard as an explicit opt-in to live mode with dry run as the default. Re-run the preview and confirm zero mutating API calls before anything goes live again.

Verbatim signal from the incident:

[2026-10-07 02:14:03] DRY-RUN: would terminate i-0a1b2c3d4e5f60718
[2026-10-07 02:14:04] DRY-RUN: would terminate i-0a1b2c3d4e5f60719

The log said preview. CloudTrail showed real TerminateInstances calls one second later.

1. Disable every trigger that can fire the script, EventBridge rules and cron entries alike, so it cannot re-fire while you investigate. Command: aws events disable-rule --name [RULE-NAME] --region [REGION] Expected: the rule state reads Disabled on describe-rule.

2. Pull the real blast list from CloudTrail for the run window, one region at a time since lookup is regional. Command: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances --start-time 2026-10-07T02:00:00Z --end-time 2026-10-07T03:00:00Z --region [REGION] Expected: a concrete list of affected resource IDs with timestamps, which you diff against the preview list in the logs.

3. Repeat step 2 for StopInstances, DeregisterImage, DeleteSnapshot, and DeleteVolume in every region the script touches. Expected: no region left unexamined, so the blast list is complete.

4. Find the inversion in the flag parsing. The classic shape is a default-live variable combined with a flag that negates it a second time, or an argparse recipe copied wrong. Rewrite it so live mode requires an explicit flag and everything else previews:

   parser.add_argument("--live", action="store_true", help="actually perform the changes")
   args = parser.parse_args()
   live_mode = args.live

   Expected: running with no flags can only ever preview.

5. Run the fixed script with no flags and watch CloudTrail for the new run window. Expected: the plan prints, and lookup-events returns zero mutating calls.

6. Restore what was destroyed from snapshots, AMIs, or backups, then re-enable the schedule only after a witnessed clean preview. Expected: restored resources verified running before the automation is re-armed.

Use this when

• Resources were deleted, stopped, or resized during a run that was supposed to preview only
• Logs claim dry-run or preview mode but CloudTrail shows TerminateInstances, StopInstances, or Delete calls in the same window
• The flag uses a double negative, a store_false recipe, or an inherited boolean you do not trust
• You inherited a cleanup script and need to prove the preview path is safe before scheduling it

Not for this skill when

• The error is AWS's own DryRunOperation response to a DryRun parameter, which is the API confirming permissions, not a script bug
• Resources were deleted by a different automation, a human in the console, or an expired lifecycle policy
• The script failed closed and changed nothing, which is a permissions issue instead

Variant phrasings

• dry run deleted resources anyway
• script ran in live mode despite the dry run flag
• --dry-run flag was ignored and instances got terminated
• preview mode deleted EC2 instances

Why it happens

Boolean plumbing with two negatives reads correctly and runs wrong. A variable named live defaults to true, a --dry-run flag sets a second variable, and the guard combines them in a way that cancels out. Copy-pasted argparse recipes make it worse: action="store_false" on a flag already named as a negative flips the meaning silently, and nothing in the log output reveals it because the log line prints before the branch.

Edge cases

• CloudTrail lookup-events covers only the last 90 days and is per region, so a cross-region script needs one query per region
• The script may have two triggers, such as an EventBridge rule plus a cron entry on an instance, and disabling one still leaves the other live
• Some deletions are irreversible: terminated instance store volumes and deleted snapshots with no AMI are gone, so restore planning starts from backups, not from wishful thinking
• A second inverted flag can hide in a helper module the main script imports, so grep the whole repo for dry_run, preview, and live before declaring it fixed

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Thread query: agent's dry-run flag was inverted - it ran the deletion plan it was supposed to preview

TL;DR: