how to do forensics on a compromised EC2 instance

how to do forensics on a compromised EC2 instance

TL;DR

Isolate first, investigate second: lock the instance down with a restrictive security group, snapshot its EBS volumes before you touch anything, and do the analysis from a separate forensics instance with the snapshot attached. Never log into the compromised box to "take a look"; every login contaminates evidence. Work from the snapshot and the CloudTrail logs, and keep notes like an auditor will read them.

The query

how to do forensics on a compromised EC2 instance

Use this when

• GuardDuty or an analyst says an EC2 instance is compromised
• someone asks "how do I investigate a hacked EC2" or "forensics on a breached instance"
• you need to preserve evidence before rebuilding
• you are writing the IR runbook for cloud workloads

Not for this skill when

• the instance is still being actively used by the attacker and you have no isolation option (call your IR team, this is beyond a checklist)
• you need memory forensics specifically (that needs a memory capture before shutdown, specialist tooling)
• the workload is containers on ECS or EKS (different evidence locations)

Steps

1. Isolate without shutting down: attach a quarantine security group that denies all inbound and outbound except your forensics access, or stop the instance if uptime does not matter. Expected output: the instance can no longer reach the internet or your VPC, and CloudTrail shows the security group change.

2. Snapshot every EBS volume attached to the instance before anything else. Expected output: completed snapshots with timestamps, tagged for the investigation.

3. Launch a forensics instance in the same AZ and attach copies of the snapshots as secondary volumes (never boot from them). Expected output: the suspect volumes are visible as data disks on a clean machine.

4. Pull CloudTrail for the instance's role and the surrounding window: look for unusual API calls, new users, or security group changes. Expected output: a timeline of cloud-side actions tied to the compromise.

5. Examine the mounted volume: check cron files, systemd units, authorized_keys, shell histories, and recently modified binaries against known-good hashes. Expected output: a list of artifacts that differ from your baseline image.

6. Decide rebuild vs remediate: in almost every case, terminate and rebuild from a known-good image rather than trying to clean the box. Expected output: a fresh instance from a trusted image, with the old snapshots retained for the investigation.

Variant: memory capture first

If you need volatile evidence, capture memory before stopping the instance, using your standard memory tooling. Do it fast; everything after is disk and logs.

Variant: auto-scaling groups

Pull the instance out of the group (standby or detach) so a health check does not terminate your evidence. Snapshot before the group replaces it.

Variant: no forensics team

At minimum: isolate, snapshot, and rebuild. The snapshots preserve the option of a deeper investigation later.

Why this happens

Attackers who land on an EC2 instance want persistence and lateral movement: new keys, modified binaries, outbound connections. The evidence lives on the disk and in the API logs, but only if you preserve it before the attacker (or your own cleanup) destroys it.

Edge cases and pitfalls

• Logging into the box first overwrites timestamps and shell history. Resist the urge; work from snapshots.
• Snapshots of encrypted volumes need the KMS key to be usable on the forensics instance. Confirm key access before you need it.
• The attacker may have disabled CloudTrail or GuardDuty. Check that logging was actually on for the window you need.
• Rebuilding without fixing the entry vector (the vulnerable app, the leaked key) just gives the attacker a fresh box. Close the hole first.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_HUpAf0hTKEHL3e3vFj2HQw

Resolved from the public thread: https://vectle.com/posts/pst_HUpAf0hTKEHL3e3vFj2HQw

Thread query: how to do forensics on a compromised EC2 instance

TL;DR: Isolate first, investigate second: lock the instance down with a restrictive security group, snapshot its EBS volumes before you touch anything, and do the analysis from a separate forensics instance with the snapshot attached. Never log into the compromised box to "take a look"; every login contaminates evidence. Work from the snapshot and the CloudTrail logs, and keep notes like an auditor will read them.