ransomware backups that actually restore

ransomware backups that actually restore

TL;DR

A backup you have never restored is a hope, not a backup. Keep three copies on two different media with one offline or immutable, and prove the restore works with regular test restores. Ransomware goes after backups first, so the copy the attacker cannot reach or rewrite is the one that saves you.

ransomware backups that actually restore

Use this when

• You are setting up backups with ransomware in mind
• Leadership asks "could we recover from ransomware" and you need an honest answer
• Your current backups live on the same network as production
• You need to justify an offline or immutable backup copy

Not for this skill when

• You need a general backup tool comparison (different skill)
• You are mid-incident with encrypted systems (follow your incident response plan first)
• You need legal advice on ransom payment (talk to counsel, not this skill)
• You want backup performance tuning (not covered here)

Steps

1. Get to 3-2-1 with one untouchable copy

Three copies of data, on two different media, one of them offline or immutable. The immutable copy is the ransomware answer: object-locked storage or a physically disconnected drive that no credential in your environment can rewrite.

restic -r [backup-repo] snapshots

Expected: restic lists snapshots across your repos, including the immutable one. If every copy is reachable with the same admin credentials, you have one copy with extra steps; fix that before anything else. (restic 0.16+; borgbackup and rclone follow the same pattern.)

2. Make the backup credentials least-privilege

The backup job should be able to write new backups and nothing else: no delete, no overwrite, no access to the immutable copy's lock settings. Ransomware encrypts whatever the compromised credentials can touch.

aws iam get-role-policy --role-name [backup-role] --policy-name [policy-name] --query PolicyDocument.Statement[].Action

Expected: the actions list shows write and read but no delete and no lock-configuration changes. If you see a wildcard or delete actions, tighten the policy. (AWS CLI v2; every cloud has an equivalent policy inspection.)

3. Run a real test restore on a schedule

Automate a restore of a meaningful subset into an isolated location and verify the files open. Monthly at minimum; weekly for critical systems.

restic -r [backup-repo] restore latest --target /tmp/restore-test-[date] --include "/srv/app/data"

Expected: the restore completes and the files in the target are intact and openable. Log the result with the date; a restore test nobody records is a restore test that did not happen. Rotate what you restore so you are not always testing the same easy files.

4. Document the recovery order and RTO

Write down what gets restored first, who does it, and how long each piece should take. Identity systems and backup infrastructure come before the app tier; networking and DNS before everything.

printf 'recovery order: 1 network/dns, 2 identity, 3 backup infra, 4 databases, 5 app tier\n' | tee recovery-order.txt

Expected: a one-page recovery order with owners and time targets that someone who has never done it could follow at 3am. If the doc lives only in one person's head, it does not exist.

5. Drill it once a year, tabletop at minimum

A full rehearsal finds the gaps the checklist misses: the restore key is with someone on vacation, the immutable copy's retention expired, the runbook references decommissioned hosts.

echo "restore drill $(date -u +%F): restored [system], [N] files verified, issues: [list]" | tee -a restore-drill-log.txt

Expected: one log line per drill with issues listed, and every issue gets an owner and a fix date. The drill is not the goal; the fixed gaps are.

Variant: cloud-native backups

Same rules, cloud-shaped: enable object lock on the backup bucket, keep backup admin roles separate from production admin roles, and test restores into an isolated account. A snapshot in the same account as production is convenient and exactly what ransomware wants.

Variant: small team with one NAS

The NAS is not the offline copy if it is on the network. Add a rotating pair of external drives, one always offsite and disconnected, or an immutable cloud copy. Test the restore from the offline copy specifically; that is the one you will need.

Variant: database backups

Restore to a scratch instance and run the app's smoke tests against it; do not just check that the dump file exists. A corrupt dump that restores zero rows is worse than no backup because it burns the recovery window.

Why this happens

Most backup setups are designed for "oops I deleted a file", not for an adversary actively trying to destroy recoverability. Ransomware operators learned years ago to find and encrypt or delete backups first, including network shares, cloud buckets reachable with the same credentials, and backup admin consoles. The 3-2-1 rule predates ransomware but answers it exactly: the offline or immutable copy is outside the attacker's reach by construction, not by hope.

Edge cases and pitfalls

• Immutability with too-short retention: a 7-day lock does not help against ransomware that dwells for 30 days before detonating; size retention to your detection lag.
• Backup software itself as attack surface: patch it and isolate its management console.
• Encrypted backups with the key stored next to them: the attacker encrypts the key too; keep restore keys offline.
• "Backup succeeded" emails without restore tests: success means bytes were written, not that they can be read back.
• Versioned cloud storage is not immutable storage; versioning can be suspended by anyone with the right permission.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_jIeoMdT-cWT0frQCgey3Bg