agent enforced mandatory cost-center tags by deleting untagged resources - it wiped a contractor's unregistered but...

TL;DR: Disable the enforcement automation immediately, then rebuild the deletion list from CloudTrail and restore from snapshots or backups. Deleting untagged resources is never an acceptable tagging strategy: replace it with tag-on-create guardrails plus a quarantine-and-notify flow with a grace period. Untagged does not mean unowned.

The CloudTrail record:

EventName: TerminateInstances  EventSource: ec2.amazonaws.com
UserAgent: tagging-enforcement-lambda  Resources: i-0a1b2c3d4e5f60718 (prod contractor VM)

1. Disable the enforcement Lambda, Config rule, or EventBridge trigger right now so it cannot delete anything else. Expected: the rule shows disabled and no new enforcement invocations appear in logs.

2. Build the full deletion list from CloudTrail for the enforcement identity over its whole active window, across all regions. Expected: every resource the automation touched, with timestamps.

3. Restore: relaunch terminated instances from their AMIs or snapshots, restore deleted volumes, and re-create anything with no backup from change records. Expected: production workloads verified running again before any policy discussion resumes.

4. Replace the delete policy with non-destructive enforcement: require tags at creation time through tag policies and service control policies that deny untagged creates, and auto-apply default tags where the creator omitted them. Expected: new resources arrive tagged, and nothing is ever deleted for a missing tag.

5. For existing untagged resources, use quarantine plus notification: flag them, notify the owner, and only after a documented grace period with no response should a human decide. Expected: a paper trail for every remediation action, with deletion as a human decision, never an automated one.

Use this when

• A tagging Lambda, Config rule, or policy deleted resources for missing tags
• CloudTrail shows deletions by an automation identity rather than a human
• Someone proposes delete-on-untagged as a governance policy
• A contractor or shadow-IT resource was destroyed by enforcement

Not for this skill when

• Resources were deleted for a real reason, such as age-based lifecycle or cost cleanup, and tags were only used for selection
• The automation only flags or quarantines untagged resources without deleting them
• You are designing tag policies for creation-time enforcement, which is the correct approach and needs no incident response

Variant phrasings

• lambda deleted untagged EC2 instances
• mandatory tagging policy deleted production resources
• config rule remediated by terminating instances
• enforce cost center tags without deleting resources

Why it happens

Someone wrote "untagged equals unmanaged equals safe to delete" into a runbook, and the automation executed it literally. Tagging coverage is never complete: contractors, acquisitions, and break-glass resources exist outside the tagging pipeline, and a silent tagging failure at creation looks identical to an abandoned resource. The automation cannot tell the difference, so it must not be given a delete action.

Edge cases

• The enforcement may have deleted resources in accounts or regions you have not checked yet; the CloudTrail sweep must cover the whole organization
• Backups may themselves have been deleted by the same policy if they were untagged, so verify backup integrity before promising recovery
• Service control policies that deny untagged creates can break CloudFormation stacks that do not propagate tags, so test in a sandbox OU first
• The contractor whose VM was wiped may have legal or contractual standing; loop in the account owner before treating it as purely technical

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_kweMV-gopyuDXeSghmCMpg