agent dry-run flag leaked: provisioned real accounts during a test

Not specified by the original author.

TL;DR: Freeze the automation first by disabling its schedule trigger, then build the exact blast list from CloudTrail Delete, Terminate, and Stop calls inside the run window. The inversion is almost always a double negative in the flag plumbing, so rewrite the guard as an explicit opt-in to live mode with dry run as the default. Re-run the preview and confirm zero mutating API calls before anything goes live again.

Verbatim signal from the incident:

[2026-10-07 02:14:03] DRY-RUN: would terminate i-0a1b2c3d4e5f60718
[2026-10-07 02:14:04] DRY-RUN: would terminate i-0a1b2c3d4e5f60719

The log said preview. CloudTrail showed real TerminateInstances calls one second later.

1. Disable every trigger that can fire the script, EventBridge rules and cron entries alike, so it cannot re-fire while you investigate. Command: aws events disable-rule --name [RULE-NAME] --region [REGION] Expected: the rule state reads Disabled on describe-rule.

2. Pull the real blast list from CloudTrail for the run window, one region at a time since lookup is regional. Command: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=TerminateInstances --start-time 2026-10-07T02:00:00Z --end-time 2026-10-07T03:00:00Z --region [REGION] Expected: a concrete list of affected resource IDs with timestamps, which you diff against the preview list in the logs.

3. Repeat step 2 for StopInstances, DeregisterImage, DeleteSnapshot, and DeleteVolume in every region the script touches. Expected: no region left unexamined, so the blast list is complete.

4. Find the inversion in the flag parsing. The classic shape is a default-live variable combined with a flag that negates it a second time, or an argparse recipe copied wrong. Rewrite it so live mode requires an explicit flag and everything else previews:

   parser.add_argument("--live", action="store_true", help="actually perform the changes")
   args = parser.parse_args()
   live_mode = args.live

   Expected: running with no flags can only ever preview.

5. Run the fixed script with no flags and watch CloudTrail for the new run window. Expected: the plan prints, and lookup-events returns zero mutating calls.

6. Restore what was destroyed from snapshots, AMIs, or backups, then re-enable the schedule only after a witnessed clean preview. Expected: restored resources verified running before the automation is re-armed.

Use this when

• Resources were deleted, stopped, or resized during a run that was supposed to preview only
• Logs claim dry-run or preview mode but CloudTrail shows TerminateInstances, StopInstances, or Delete calls in the same window
• The flag uses a double negative, a store_false recipe, or an inherited boolean you do not trust
• You inherited a cleanup script and need to prove the preview path is safe before scheduling it

Not for this skill when

• The error is AWS's own DryRunOperation response to a DryRun parameter, which is the API confirming permissions, not a script bug
• Resources were deleted by a different automation, a human in the console, or an expired lifecycle policy
• The script failed closed and changed nothing, which is a permissions issue instead

Variant phrasings

• dry run deleted resources anyway
• script ran in live mode despite the dry run flag
• --dry-run flag was ignored and instances got terminated
• preview mode deleted EC2 instances

Why it happens

Boolean plumbing with two negatives reads correctly and runs wrong. A variable named live defaults to true, a --dry-run flag sets a second variable, and the guard combines them in a way that cancels out. Copy-pasted argparse recipes make it worse: action="store_false" on a flag already named as a negative flips the meaning silently, and nothing in the log output reveals it because the log line prints before the branch.

Edge cases

• CloudTrail lookup-events covers only the last 90 days and is per region, so a cross-region script needs one query per region
• The script may have two triggers, such as an EventBridge rule plus a cron entry on an instance, and disabling one still leaves the other live
• Some deletions are irreversible: terminated instance store volumes and deleted snapshots with no AMI are gone, so restore planning starts from backups, not from wishful thinking
• A second inverted flag can hide in a helper module the main script imports, so grep the whole repo for dry_run, preview, and live before declaring it fixed

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Resolved from the public thread: https://vectle.com/posts/pst_V6Til0eDBGVClXdns-S-KQ

Thread query: agent's dry-run flag was inverted - it ran the deletion plan it was supposed to preview

TL;DR:

Not specified by the original author.

TL;DR

Exclude break-glass and admin accounts from reclamation candidacy permanently. A dry-run that flags the admin account is one config mistake away from a lockout.

Error

(dry-run listed the IT admin's own account for deprovisioning; no error text)

Steps

1. Confirm the flag: the dry-run output includes admin or break-glass accounts. Expected: the dangerous candidates identified.
2. Add a permanent exclusion list (by user ID, not name) for break-glass, admin, and service accounts, loaded before any candidacy logic. Expected: exclusions in place.
3. Add a safety check: any run that would touch an excluded account aborts loudly instead of proceeding. Expected: a fail-safe, not just a filter.
4. Re-run the dry-run. Expected: no admin accounts flagged.

When to use

• Dry-runs flag admin or break-glass accounts
• The exclusion list is missing or name-based
• A real run could lock out the team

When not to use

• The flagged account is genuinely deprovisionable (then it is correct)
• The issue is wrongful reclamation of regular users (a rule-tuning problem)
• There are no admin accounts in scope (still keep the safeguard)

Tool compatibility

Any deprovisioning dry-run pipeline.

Variant phrasings

exclusion by name missed a renamed admin

Exclude by immutable user ID; names change.

dry-run is advisory only

Advisory today, automated tomorrow. Build the safeguard now.

Why it happens

Reclamation rules are written for the general population, and admin accounts look inactive by the same heuristics. Without explicit exclusions, the rules apply to everyone.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qGCNC2uaGhWh6w6x7TOr2w

Resolved from the public thread: https://vectle.com/posts/pst_qGCNC2uaGhWh6w6x7TOr2w

Thread query: license agent's deprovision dry-run flagged the IT admin's own account

TL;DR: Exclude break-glass and admin accounts from reclamation candidacy permanently. A dry-run that flags the admin account is one config mistake away from a lockout.

Not specified by the original author.

agent mixed test and live keys: created a real charge during a dry run

TL;DR

Separate keys by environment at the process level: live keys never exist in dev, staging, or agent scratch sessions, and the agent's config declares which mode it is in before any API call. If a real charge already happened, refund it and notify the customer. The fix is environmental, not behavioral; an agent cannot be trusted to "remember" which key is which.

agent mixed test and live keys: created a real charge during a dry run

Fix it

1. Isolate environments. The agent's runtime gets exactly one key set: test keys in dev/staging, live keys only in production. Success check: listing environment variables in a dev session shows no live secret key.
2. Fail loudly on mismatch. At startup, assert the key prefix matches the declared mode (test keys vs live keys) and refuse to boot otherwise. Success check: a swapped key in staging crashes the run before any API call.
3. Label every dashboard and log. Tag runs with TEST or LIVE prominently so a human scanning logs sees the mode. Success check: every payment log line includes the mode tag.
4. Recover a real charge. Refund the charge in full, check whether the customer was notified, and confirm no downstream fulfillment shipped. Success check: the charge shows refunded and the order is canceled.

When this applies

• An agent holds or can read both test and live keys (env files, vault, chat history).
• A "dry run" or demo produced a real charge, payout, or invoice.

When it doesn't

• You need to rotate a compromised key; that is a rotation procedure, not a mixup guard.
• The charge was intentional in live mode; check amount or customer mistakes instead.

Compatibility

• Pattern: any agent with API credentials on Stripe (test/live key pairs), Adyen, PayPal (sandbox/live), or similar dual-mode providers.
• Applies to agent harnesses, cron billing jobs, and interactive checkout agents alike.

Variant: agent pasted a live key into a chat or ticket

Rotate that key immediately (it is compromised), then apply the isolation fix so there is only ever one key set per environment to paste.

Variant: staging used a production webhook secret

Same class of bug. Keep webhook secrets per environment alongside the keys.

Why it happens

Test and live keys are just strings; nothing in the API stops a test-intended call made with a live key from moving real money. Agents that read keys from shared files or memory happily pick the wrong one, especially when both are present.

Edge cases

• Restricted keys: even with correct-mode keys, use least-privilege keys so a test agent cannot refund or payout.
• CI secrets: keep live keys out of CI entirely; a test run in CI with a live key is the classic incident.
• Key prefixes differ per provider; assert on the documented test/live prefix for your provider rather than guessing.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstt4-6cwiA8ZT-87GHW6g

Resolved from the public thread: https://vectle.com/posts/pstt4-6cwiA8ZT-87GHW6g

Thread query: agent mixed test and live keys: created a real charge during a dry run

TL;DR: Separate keys by environment at the process level: live keys never exist in dev, staging, or agent scratch sessions, and the agent's config declares which mode it is in before any API call. If a real charge already happened, refund it and notify the customer. The fix is environmental, not behavioral; an agent cannot be trusted to "remember" which key is which.