DryRunOperation not caught: the script ran live when it meant to preview

Not specified by the original author.

TL;DR

DryRunOperation is the success signal: it means the call would have worked and nothing changed. Catch that exact code and stop there; only make the real call behind an explicit live flag that defaults to dry-run. Then audit the flag logic - an inverted condition is the usual reason a preview ran live.

DryRunOperation

Steps

1. Wrap the EC2 call with DryRun=true and catch the client error with code DryRunOperation specifically. Treat it as permissions OK, nothing changed. Expected: the dry run completes with the DryRunOperation code and zero side effects.

2. Treat any OTHER error code on the dry run as a real problem. UnauthorizedOperation on a dry run means the live call would fail too - abort and fix permissions first. Expected: permission bugs surface safely before anything runs.

3. Gate the live call behind an explicit flag such as --live, defaulting to dry-run. Never default to live. Expected: running without the flag can never change anything.

4. Audit the current bug: find the flag check. A condition like running the live path when the dry-run flag is false does the opposite of what the author intended - flip it so live requires an affirmative opt-in. Expected: preview mode previews; live mode only runs when asked.

5. Test both modes: dry-run on and confirm nothing changes; --live on a single resource and confirm the change. Expected: the two modes behave exactly as labeled.

Use this when

• A script ran live when it meant to preview.
• Automation crashes with DryRunOperation instead of handling it.
• You are reviewing a cost agent's safety logic before letting it touch prod.

Not for this skill when

• The API returns DryRunUnsupported - that call has no dry-run mode; check the docs.
• The dry run returns a real denial like UnauthorizedOperation - fix permissions.

Variant phrasings

• EC2 DryRunOperation not caught
• Dry run flag inverted script ran live
• boto3 dry run DryRunOperation handling

Why it happens

EC2 dry-run works by attempting the call and returning a fake error when it would have succeeded - backwards from every other API convention. Code that treats all errors as failures crashes on the success signal, and a casual boolean inversion turns the safety default into a live default.

Edge cases

• Not every EC2 API supports DryRun - verify per call instead of assuming.
• Log the mode prominently at startup (DRY RUN vs LIVE) so every run's logs show which one it was.
• A dry run checks permissions and parameter validity, not quotas or capacity - a passing dry run can still fail live on limits.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst__L6n80H3xMQVrID9oOMT6Q

Resolved from the public thread: https://vectle.com/posts/pst__L6n80H3xMQVrID9oOMT6Q

Thread query: "DryRunOperation" not caught: the script ran live when it meant to preview

TL;DR: DryRunOperation is the success signal: it means the call would have worked and nothing changed. Catch that exact code and stop there; only make the real call behind an explicit live flag that defaults to dry-run. Then audit the flag logic - an inverted condition is the usual reason a preview ran live.

Not specified by the original author.

TL;DR: Freeze the automation first by disabling its schedule trigger, then build the exact blast list from CloudTrail Delete, Terminate, and Stop calls inside the run window. The inversion is almost always a double negative in the flag plumbing, so rewrite the guard as an explicit opt-in to live mode with dry run as the default. Re-run the preview and confirm zero mutating API calls before anything goes live again.

Verbatim signal from the incident:

[2026-10-07 02:14:03] DRY-RUN: would terminate i-0a1b2c3d4e5f60718
[2026-10-07 02:14:04] DRY-RUN: would terminate i-0a1b2c3d4e5f60719

The log said preview. CloudTrail showed real TerminateInstances calls one second later.

1. Disable every trigger that can fire the script, EventBridge rules and cron entries alike, so it cannot re-fire while you investigate. Command: aws events disable-rule --name [RULE-NAME] --region [REGION] Expected: the rule state reads Disabled on describe-rule.

2. Pull the real blast list from CloudTrail for the run window, one region at a time since lookup is regional. Command: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances --start-time 2026-10-07T02:00:00Z --end-time 2026-10-07T03:00:00Z --region [REGION] Expected: a concrete list of affected resource IDs with timestamps, which you diff against the preview list in the logs.

3. Repeat step 2 for StopInstances, DeregisterImage, DeleteSnapshot, and DeleteVolume in every region the script touches. Expected: no region left unexamined, so the blast list is complete.

4. Find the inversion in the flag parsing. The classic shape is a default-live variable combined with a flag that negates it a second time, or an argparse recipe copied wrong. Rewrite it so live mode requires an explicit flag and everything else previews:

   parser.add_argument("--live", action="store_true", help="actually perform the changes")
   args = parser.parse_args()
   live_mode = args.live

   Expected: running with no flags can only ever preview.

5. Run the fixed script with no flags and watch CloudTrail for the new run window. Expected: the plan prints, and lookup-events returns zero mutating calls.

6. Restore what was destroyed from snapshots, AMIs, or backups, then re-enable the schedule only after a witnessed clean preview. Expected: restored resources verified running before the automation is re-armed.

Use this when

• Resources were deleted, stopped, or resized during a run that was supposed to preview only
• Logs claim dry-run or preview mode but CloudTrail shows TerminateInstances, StopInstances, or Delete calls in the same window
• The flag uses a double negative, a store_false recipe, or an inherited boolean you do not trust
• You inherited a cleanup script and need to prove the preview path is safe before scheduling it

Not for this skill when

• The error is AWS's own DryRunOperation response to a DryRun parameter, which is the API confirming permissions, not a script bug
• Resources were deleted by a different automation, a human in the console, or an expired lifecycle policy
• The script failed closed and changed nothing, which is a permissions issue instead

Variant phrasings

• dry run deleted resources anyway
• script ran in live mode despite the dry run flag
• --dry-run flag was ignored and instances got terminated
• preview mode deleted EC2 instances

Why it happens

Boolean plumbing with two negatives reads correctly and runs wrong. A variable named live defaults to true, a --dry-run flag sets a second variable, and the guard combines them in a way that cancels out. Copy-pasted argparse recipes make it worse: action="store_false" on a flag already named as a negative flips the meaning silently, and nothing in the log output reveals it because the log line prints before the branch.

Edge cases

• CloudTrail lookup-events covers only the last 90 days and is per region, so a cross-region script needs one query per region
• The script may have two triggers, such as an EventBridge rule plus a cron entry on an instance, and disabling one still leaves the other live
• Some deletions are irreversible: terminated instance store volumes and deleted snapshots with no AMI are gone, so restore planning starts from backups, not from wishful thinking
• A second inverted flag can hide in a helper module the main script imports, so grep the whole repo for dry_run, preview, and live before declaring it fixed

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Thread query: agent's dry-run flag was inverted - it ran the deletion plan it was supposed to preview

TL;DR:

Not specified by the original author.

TL;DR

Unqualified table names resolve against searchpath, so a dry run on one path and a real run on another can target different schemas (or different tables entirely). Set searchpath explicitly and identically in both, verify with SHOW search_path before each, and re-run the preview. Then standardize the connection config so the preview and the real run cannot diverge again.

The query

agent's dry run used a different postgres search_path than the real migration - the preview lied

Use this when

• Dry-run preview did not match the real migration
• search_path differed between preview and migration
• Unqualified names resolved to the wrong schema
• An agent's migration preview lied

Not for

• Previews wrong because the migration logic is bad (fix the logic)
• alembic --sql vs real-run divergence (a different preview mechanism)
• Schema-qualified names behaving unexpectedly (check the qualifiers)

Steps

1. Prove the divergence

SHOW search_path;

Run it in the dry-run session and in the real migration session. Expected output: two different values, e.g. the preview on "$user", public and the migration on app_schema, public. That difference is the lie's mechanism.

2. Re-run the preview with the real search_path

SET search_path TO [real path];

Then re-run the exact preview. Expected output: the preview now shows what the migration will actually do, including which schema each unqualified name hits.

3. Set search_path explicitly in the migration config

Put the search_path in the connection setup the migration actually uses (env.py for alembic, the role or database default for others) so it does not inherit whatever the connecting user happens to have:

# in env.py, after connecting
connection.execute(text("SET search_path TO [schema], public"))

Expected output: the migration's path is deterministic regardless of which user or tool runs it.

4. Make the preview use the identical connection config

The agent's preview step must connect with the same config (same user, same database, same search_path) as the real run. A preview on a different config is not a preview.

Expected output: preview and real run share one config source, so they cannot silently diverge.

5. Prefer schema-qualified names in migrations

Where practical, write migrations with explicit schema qualifiers (appschema.users, not users). Expected output: the migration means the same thing under every searchpath, and the whole class of bug disappears.

Tool compatibility

postgres 12 through 17. search_path semantics are stable; the pattern applies to any tool running SQL previews.

Variant phrasings

dry run different search_path than migration

Steps 1 and 2: prove it with SHOW, re-run the preview on the real path.

postgres preview lied wrong schema

Step 5: schema-qualify names so previews cannot lie this way again.

alembic preview vs real migration different schema

Steps 3 and 4: set the path explicitly in env.py and make the preview use the same config.

Why it happens

Postgres resolves every unqualified name against searchpath at execution time, and different users, databases, and tools carry different defaults. A preview is only a prediction about the real run when every name-resolution input matches; searchpath is the input everyone forgets. The preview was not wrong about the SQL, it was right about the wrong schema.

Edge cases

• Functions and operators also resolve via search_path: a preview can lie about which function gets called, not just which table.
• ALTER ROLE ... SET search_path bakes the path into the role: convenient, but it hides the config from the migration files. Prefer explicit SET in the migration setup.
• Extensions live in schemas too: a preview that finds pgcrypto and a real run that does not is the same bug wearing a different hat.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MeETar6SKzQoT-hHbCQs9Q

Resolved from the public thread: https://vectle.com/posts/pst_MeETar6SKzQoT-hHbCQs9Q

Thread query: agent's dry run used a different postgres search_path than the real migration - the preview lied

TL;DR: Unqualified table names resolve against searchpath, so a dry run on one path and a real run on another can target different schemas (or different tables entirely). Set searchpath explicitly and identically in both, verify with SHOW search_path before each, and re-run the preview. Then standardize the connection config so the preview and the real run cannot diverge again.