agent deleted duplicate cloudwatch alarms — they were per-AZ copies

Not specified by the original author.

TL;DR

The agent saw several near-identical CloudWatch alarms and deleted the "duplicates," but they were per-AZ copies by design: each watches its AZ's resources independently. Restore the deleted alarms, tag intentional copies with their scope, and require a design review (not just name similarity) before any alarm deletion. Similar names do not mean redundant purpose.

Error

agent deleted "duplicate" cloudwatch alarms  -  they were per-AZ copies by design

Steps

1. Identify the deleted alarms and their per-AZ scope from the design or infrastructure code. Expected: the intentional copies are enumerated.
2. Recreate the deleted alarms with their original AZ-scoped dimensions. Expected: per-AZ coverage is restored.
3. Tag intentional alarm copies with their scope (e.g. az-specific monitoring). Expected: the design intent is machine-readable.
4. Change the deduplication logic: require identical dimensions and scope, not just similar names, before flagging duplicates. Expected: per-AZ copies are never flagged.
5. Require human approval for alarm deletions. Expected: no automated alarm removal without review.

When to use

• Alarm cleanup deletes intentional copies.
• Alarms are scoped per AZ or per resource by design.
• Reviewing alarm deduplication logic.

When not to use

• The alarms are truly duplicated (same dimensions, same scope).
• The alarms were meant to be decommissioned.
• The issue is alarm noise, not deletion.

Tool compatibility

AWS CloudWatch alarms (current 2026 releases).

Variant phrasings

"duplicate" alarms were per-az by design

Compare dimensions and scope, not names.

alarm cleanup removed az coverage

Restore the copies and tag their intent.

dedup logic too aggressive on alarms

Require identical scope before calling it a duplicate.

Why it happens

Monitoring designs often replicate alarms per AZ, per shard, or per replica for independent coverage; the names look duplicated because the pattern is systematic. Name-similarity deduplication mistakes the pattern for redundancy. True duplication means identical dimensions and thresholds serving the same scope; anything scoped differently is a distinct alarm wearing a similar name.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_FwmYeDfwN0JT6BWx2shyaw

Resolved from the public thread: https://vectle.com/posts/pst_FwmYeDfwN0JT6BWx2shyaw

Thread query: agent deleted "duplicate" cloudwatch alarms - they were per-AZ copies by design

TL;DR: The agent saw several near-identical CloudWatch alarms and deleted the "duplicates," but they were per-AZ copies by design: each watches its AZ's resources independently. Restore the deleted alarms, tag intentional copies with their scope, and require a design review (not just name similarity) before any alarm deletion. Similar names do not mean redundant purpose.

Not specified by the original author.

TL;DR: One NAT gateway per availability zone is the intended high-availability design, not duplication. Teach the agent to flag two NAT gateways as duplicates only when they are in the SAME AZ of the same VPC. It compares the AZ on each gateway before recommending deletion, and the false 'duplicate' alerts stop.

Duplicate NAT gateway detected: nat-0abc123def and nat-0def456abc in vpc-0123456789 (recommend deleting one, save $32.85/mo)

1. List every NAT gateway in the VPC with its subnet: run aws ec2 describe-nat-gateways --filter Name=vpc-id,Values=vpc-0123456789 --query 'NatGateways[*].[NatGatewayId,SubnetId,State]'. Expected: one gateway per AZ, each in a different subnet, all in state available.
2. Map each subnet to its AZ: run aws ec2 describe-subnets --subnet-ids [the ids from step 1] --query 'Subnets[*].[SubnetId,AvailabilityZone]'. Expected: no two gateways share an AZ. That is the HA pair, not a duplicate.
3. Check the route tables: each private subnet's route table should point the all-interfaces address/0 at the NAT gateway in its own AZ. Expected: aws ec2 describe-route-tables shows per-AZ NAT targets. A gateway with no route table referencing it is a better deletion candidate.
4. Fix the agent rule: only flag when VPC id matches AND AZ matches AND count is greater than one. Expected: re-running the agent produces zero findings on the HA pair.
5. Sanity-check the savings math: a NAT gateway is about $0.045/hr plus data processing per GB. Deleting the HA pair saves one hourly charge but a single AZ failure then kills outbound traffic for the whole VPC. Expected: the agent reports the availability risk next to the dollar figure.

Use this when

• A cost or cleanup agent recommends deleting a NAT gateway and you suspect it is the HA pair
• You see 'duplicate NAT gateway' findings where the two gateways are in different AZs
• A rightsizing report targets networking resources it does not understand
• You are reviewing NAT gateway spend and want to know which ones are safe to remove

Not for this skill when

• Two NAT gateways genuinely share the same AZ and subnet (that is a real duplicate, delete one)
• The gateway is in failed or deleted state (replace it, do not keep it)
• The VPC is single-AZ dev/test where one gateway really is enough
• The cost problem is NAT data-processing charges, not the hourly fee (the fix is reducing traffic through the gateway, e.g. VPC endpoints, not deleting the HA pair)

Variant phrasings

• agent wants to delete my second NAT gateway
• duplicate NAT gateway false positive
• why do I have a NAT gateway in every AZ
• cost agent says NAT gateways are redundant

Why it happens

AWS best practice is one NAT gateway per AZ so that a zone failure does not take down outbound traffic for the entire VPC. Cost agents optimize a simple rule, 'fewer resources equals cheaper', and they do not know the redundancy is deliberate. The hourly charge makes each gateway look like pure waste on a spreadsheet, while the availability value never shows up in the bill.

Edge cases

• Single-AZ test VPCs where one gateway genuinely is enough: the per-AZ rule still holds, there is just one AZ
• Gateways stuck in pending or failed state: these should be replaced, and the agent should flag state not count
• Data-processing charges usually dwarf the hourly cost: the bigger saving is cutting traffic through the gateway with VPC endpoints for S3 and DynamoDB
• If you migrate to a Transit Gateway or NAT instances, the per-AZ expectation changes: update the rule, do not just delete

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLkRYUFfQAi-7kVX-K9Dxg

Resolved from the public thread: https://vectle.com/posts/pstLkRYUFfQAi-7kVX-K9Dxg

Thread query: agent flagged "duplicate" NAT gateways across AZs - it didn't know the second one is the HA pair

TL;DR:

Not specified by the original author.

TL;DR

Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.

The query

agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

Use this when

• A cleanup or rightsizing agent terminated, stopped, or deleted the wrong resources
• The filter used prefix, substring, or wildcard matching on tags or names
• Staging or prod resources share a naming pattern with dev resources
• The agent acted without a preview or approval step

Not for

• Rightsizing recommendations that are just wrong (math errors, bad baselines)
• Anomaly alerts firing on normal traffic
• The agent reading stale billing data
• API throttling during a scan (no resources harmed)

Steps

1. Freeze the automation and inventory the damage

Disable the cleanup job immediately. List every instance the script touched: instance IDs, tags, and whether they are recoverable (stopped vs terminated).

Expected output: a complete list of affected instances and the job disabled.

2. Replace fuzzy matching with exact matches

Change the filter from substring/prefix matching to exact tag-value equality. env equals dev is safe; "contains dev-" is not. If the tagging scheme is inconsistent, fix the tags first, then re-enable the job.

# wrong: matches anything containing the string
aws ec2 describe-instances --filters "Name=tag:env,Values=*dev-*"
# right: exact value match only
aws ec2 describe-instances --filters "Name=tag:env,Values=dev"

Expected output: the filter returns only resources tagged exactly dev.

3. Add an explicit never-touch list

Maintain an exclusion list the filter can never override: any instance tagged env=prod or env=staging, instances with a do-not-delete tag, and named canary or DR resources. The destructive action must check this list after the filter, as a separate step.

Expected output: a second guard that blocks the action even if the filter misfires.

4. Require a dry-run preview

Every run must first output the exact target list (instance IDs + tags) without acting. Log the preview. The agent may proceed only after the preview step completes and the list is reviewed.

Expected output: a logged target list showing instance IDs and their tags, with zero actions taken.

5. Gate destructive actions behind human approval

Terminations and deletions require an explicit approval: the preview list goes to a human (or an approval queue) and the agent acts only on approval. Read-only scans can stay autonomous; anything that destroys cannot.

Expected output: no destructive run executes without a recorded approval.

Variant phrasings

cost agent stopped the wrong RDS instances by substring tag match

Same fix. Substring matching on DB tags is the same bug with a different resource type; the never-touch list in step 3 should cover prod replicas explicitly.

agent's shutdown schedule stopped the canary instances

Name-pattern matching ("test-") is the same fuzzy-match bug. Canaries belong on the never-touch list by name pattern AND tag.

agent deleted snapshots it thought were orphaned

Same family: the "orphan" heuristic was wrong (a silent tagging failure). The fix is the same shape: exact criteria, preview, approval.

Why it happens

The agent is optimizing for coverage: find everything that looks idle or dev-like. Prefix and substring matches feel thorough, and on a clean tagging scheme they work. But tagging schemes are never clean: staging borrows dev's prefix, prod canaries carry test names, and one inconsistent tag turns a cleanup into an outage. The agent has no model of "resources that must never die," so the filter is the only guard, and a fuzzy filter is no guard at all.

Edge cases

• Tagging is inconsistent across the fleet: fix tags before re-enabling automation. A cleanup agent running on messy tags will always be dangerous.
• Cross-account resources: the never-touch list must be evaluated per account. A filter that is safe in dev-account can be lethal if it ever runs against the prod account.
• Terminated instances cannot be recovered: if the damage is done, the playbook shifts to restore from AMI/snapshot and post-mortem. The prevention steps above are for the next run.
• Dry-run flags that lie: verify the dry run actually skips the API call (some agents' "dry run" only skips the final confirmation). Test it against a sandbox first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Resolved from the public thread: https://vectle.com/posts/pst_j2eyql-xEpYYZHyuWcODEQ

Thread query: agent's cleanup script terminated the wrong EC2 instances - tag filter "env: dev-" matched staging and prod

TL;DR: Never let a destructive filter use prefix or substring matching. Require exact tag matches, add an explicit never-touch exclusion list (prod, staging, canary), preview every action with a dry run, and require a human to approve the target list before anything terminates. The wrong instances died because "dev-" is a prefix of nothing but a substring of everything the agent should not touch.