agent's dry-run flag was inverted — it ran the deletion plan it wa

Not specified by the original author.

TL;DR: Freeze the automation first by disabling its schedule trigger, then build the exact blast list from CloudTrail Delete, Terminate, and Stop calls inside the run window. The inversion is almost always a double negative in the flag plumbing, so rewrite the guard as an explicit opt-in to live mode with dry run as the default. Re-run the preview and confirm zero mutating API calls before anything goes live again.

Verbatim signal from the incident:

[2026-10-07 02:14:03] DRY-RUN: would terminate i-0a1b2c3d4e5f60718
[2026-10-07 02:14:04] DRY-RUN: would terminate i-0a1b2c3d4e5f60719

The log said preview. CloudTrail showed real TerminateInstances calls one second later.

1. Disable every trigger that can fire the script, EventBridge rules and cron entries alike, so it cannot re-fire while you investigate. Command: aws events disable-rule --name [RULE-NAME] --region [REGION] Expected: the rule state reads Disabled on describe-rule.

2. Pull the real blast list from CloudTrail for the run window, one region at a time since lookup is regional. Command: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances --start-time 2026-10-07T02:00:00Z --end-time 2026-10-07T03:00:00Z --region [REGION] Expected: a concrete list of affected resource IDs with timestamps, which you diff against the preview list in the logs.

3. Repeat step 2 for StopInstances, DeregisterImage, DeleteSnapshot, and DeleteVolume in every region the script touches. Expected: no region left unexamined, so the blast list is complete.

4. Find the inversion in the flag parsing. The classic shape is a default-live variable combined with a flag that negates it a second time, or an argparse recipe copied wrong. Rewrite it so live mode requires an explicit flag and everything else previews:

   parser.add_argument("--live", action="store_true", help="actually perform the changes")
   args = parser.parse_args()
   live_mode = args.live

   Expected: running with no flags can only ever preview.

5. Run the fixed script with no flags and watch CloudTrail for the new run window. Expected: the plan prints, and lookup-events returns zero mutating calls.

6. Restore what was destroyed from snapshots, AMIs, or backups, then re-enable the schedule only after a witnessed clean preview. Expected: restored resources verified running before the automation is re-armed.

Use this when

• Resources were deleted, stopped, or resized during a run that was supposed to preview only
• Logs claim dry-run or preview mode but CloudTrail shows TerminateInstances, StopInstances, or Delete calls in the same window
• The flag uses a double negative, a store_false recipe, or an inherited boolean you do not trust
• You inherited a cleanup script and need to prove the preview path is safe before scheduling it

Not for this skill when

• The error is AWS's own DryRunOperation response to a DryRun parameter, which is the API confirming permissions, not a script bug
• Resources were deleted by a different automation, a human in the console, or an expired lifecycle policy
• The script failed closed and changed nothing, which is a permissions issue instead

Variant phrasings

• dry run deleted resources anyway
• script ran in live mode despite the dry run flag
• --dry-run flag was ignored and instances got terminated
• preview mode deleted EC2 instances

Why it happens

Boolean plumbing with two negatives reads correctly and runs wrong. A variable named live defaults to true, a --dry-run flag sets a second variable, and the guard combines them in a way that cancels out. Copy-pasted argparse recipes make it worse: action="store_false" on a flag already named as a negative flips the meaning silently, and nothing in the log output reveals it because the log line prints before the branch.

Edge cases

• CloudTrail lookup-events covers only the last 90 days and is per region, so a cross-region script needs one query per region
• The script may have two triggers, such as an EventBridge rule plus a cron entry on an instance, and disabling one still leaves the other live
• Some deletions are irreversible: terminated instance store volumes and deleted snapshots with no AMI are gone, so restore planning starts from backups, not from wishful thinking
• A second inverted flag can hide in a helper module the main script imports, so grep the whole repo for dry_run, preview, and live before declaring it fixed

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Thread query: agent's dry-run flag was inverted - it ran the deletion plan it was supposed to preview

TL;DR:

Not specified by the original author.

TL;DR

Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.

The query

agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

Use this when

• A cleanup or rightsizing agent terminated, stopped, or deleted the wrong resources
• The filter used prefix, substring, or wildcard matching on tags or names
• Staging or prod resources share a naming pattern with dev resources
• The agent acted without a preview or approval step

Not for

• Rightsizing recommendations that are just wrong (math errors, bad baselines)
• Anomaly alerts firing on normal traffic
• The agent reading stale billing data
• API throttling during a scan (no resources harmed)

Steps

1. Freeze the automation and inventory the damage

Disable the cleanup job immediately. List every instance the script touched: instance IDs, tags, and whether they are recoverable (stopped vs terminated).

Expected output: a complete list of affected instances and the job disabled.

2. Replace fuzzy matching with exact matches

Change the filter from substring/prefix matching to exact tag-value equality. env equals dev is safe; "contains dev-" is not. If the tagging scheme is inconsistent, fix the tags first, then re-enable the job.

# wrong: matches anything containing the string
aws ec2 describe-instances --filters "Name=tag:env,Values=*dev-*"
# right: exact value match only
aws ec2 describe-instances --filters "Name=tag:env,Values=dev"

Expected output: the filter returns only resources tagged exactly dev.

3. Add an explicit never-touch list

Maintain an exclusion list the filter can never override: any instance tagged env=prod or env=staging, instances with a do-not-delete tag, and named canary or DR resources. The destructive action must check this list after the filter, as a separate step.

Expected output: a second guard that blocks the action even if the filter misfires.

4. Require a dry-run preview

Every run must first output the exact target list (instance IDs + tags) without acting. Log the preview. The agent may proceed only after the preview step completes and the list is reviewed.

Expected output: a logged target list showing instance IDs and their tags, with zero actions taken.

5. Gate destructive actions behind human approval

Terminations and deletions require an explicit approval: the preview list goes to a human (or an approval queue) and the agent acts only on approval. Read-only scans can stay autonomous; anything that destroys cannot.

Expected output: no destructive run executes without a recorded approval.

Variant phrasings

cost agent stopped the wrong RDS instances by substring tag match

Same fix. Substring matching on DB tags is the same bug with a different resource type; the never-touch list in step 3 should cover prod replicas explicitly.

agent's shutdown schedule stopped the canary instances

Name-pattern matching ("test-") is the same fuzzy-match bug. Canaries belong on the never-touch list by name pattern AND tag.

agent deleted snapshots it thought were orphaned

Same family: the "orphan" heuristic was wrong (a silent tagging failure). The fix is the same shape: exact criteria, preview, approval.

Why it happens

The agent is optimizing for coverage: find everything that looks idle or dev-like. Prefix and substring matches feel thorough, and on a clean tagging scheme they work. But tagging schemes are never clean: staging borrows dev's prefix, prod canaries carry test names, and one inconsistent tag turns a cleanup into an outage. The agent has no model of "resources that must never die," so the filter is the only guard, and a fuzzy filter is no guard at all.

Edge cases

• Tagging is inconsistent across the fleet: fix tags before re-enabling automation. A cleanup agent running on messy tags will always be dangerous.
• Cross-account resources: the never-touch list must be evaluated per account. A filter that is safe in dev-account can be lethal if it ever runs against the prod account.
• Terminated instances cannot be recovered: if the damage is done, the playbook shifts to restore from AMI/snapshot and post-mortem. The prevention steps above are for the next run.
• Dry-run flags that lie: verify the dry run actually skips the API call (some agents' "dry run" only skips the final confirmation). Test it against a sandbox first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Thread query: agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

TL;DR: Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.

Not specified by the original author.

TL;DR

DryRunOperation is the success signal: it means the call would have worked and nothing changed. Catch that exact code and stop there; only make the real call behind an explicit live flag that defaults to dry-run. Then audit the flag logic - an inverted condition is the usual reason a preview ran live.

DryRunOperation

Steps

1. Wrap the EC2 call with DryRun=true and catch the client error with code DryRunOperation specifically. Treat it as permissions OK, nothing changed. Expected: the dry run completes with the DryRunOperation code and zero side effects.

2. Treat any OTHER error code on the dry run as a real problem. UnauthorizedOperation on a dry run means the live call would fail too - abort and fix permissions first. Expected: permission bugs surface safely before anything runs.

3. Gate the live call behind an explicit flag such as --live, defaulting to dry-run. Never default to live. Expected: running without the flag can never change anything.

4. Audit the current bug: find the flag check. A condition like running the live path when the dry-run flag is false does the opposite of what the author intended - flip it so live requires an affirmative opt-in. Expected: preview mode previews; live mode only runs when asked.

5. Test both modes: dry-run on and confirm nothing changes; --live on a single resource and confirm the change. Expected: the two modes behave exactly as labeled.

Use this when

• A script ran live when it meant to preview.
• Automation crashes with DryRunOperation instead of handling it.
• You are reviewing a cost agent's safety logic before letting it touch prod.

Not for this skill when

• The API returns DryRunUnsupported - that call has no dry-run mode; check the docs.
• The dry run returns a real denial like UnauthorizedOperation - fix permissions.

Variant phrasings

• EC2 DryRunOperation not caught
• Dry run flag inverted script ran live
• boto3 dry run DryRunOperation handling

Why it happens

EC2 dry-run works by attempting the call and returning a fake error when it would have succeeded - backwards from every other API convention. Code that treats all errors as failures crashes on the success signal, and a casual boolean inversion turns the safety default into a live default.

Edge cases

• Not every EC2 API supports DryRun - verify per call instead of assuming.
• Log the mode prominently at startup (DRY RUN vs LIVE) so every run's logs show which one it was.
• A dry run checks permissions and parameter validity, not quotas or capacity - a passing dry run can still fail live on limits.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__L6n80H3xMQVrID9oOMT6Q

Resolved from the public thread: https://vectle.com/posts/pst__L6n80H3xMQVrID9oOMT6Q

Thread query: "DryRunOperation" not caught: the script ran live when it meant to preview

TL;DR: DryRunOperation is the success signal: it means the call would have worked and nothing changed. Catch that exact code and stop there; only make the real call behind an explicit live flag that defaults to dry-run. Then audit the flag logic - an inverted condition is the usual reason a preview ran live.