log retention policy that survives an audit

TL;DR

An audit-surviving retention policy says three things: how long each log class is kept, where it lives, and who approved the deletion. Write it as a one-page doc, implement it with storage lifecycle rules (hot for 14-30 days, cheap archive after), and keep an approval trail for every deletion. Auditors do not demand infinite retention; they demand that what you do matches what you wrote down.

Error / query

log retention policy that survives an audit

Use this skill when

• An auditor asked for your log retention policy and you do not have one written down
• Legal or compliance gave you a retention number and nobody implemented it
• You are deleting old logs to save cost and want the deletion to be defensible
• Different teams keep logs for wildly different periods with no rationale

Not for this skill when

• You have no centralized logging yet; get aggregation working before writing policy
• The requirement is real-time monitoring, not retention; those are different problems
• You are under active litigation hold; nothing gets deleted, policy or not

Steps

Step 1: Inventory what log classes you actually have

ls /var/log/ | head -20; echo "---containers---"; crictl logs --help >/dev/null 2>&1 && echo "container runtime present" || echo "no container runtime"

Expected: a list of system logs plus confirmation of container log sources; each class gets its own retention line in the policy.

Step 2: Check current retention settings before writing policy

grep -h 'rotate\|maxage\|retention' /etc/logrotate.d/* 2>/dev/null | head -10

Expected: the existing rotation rules; note anything set to "forever" or missing entirely, those are your gaps.

Step 3: Write the policy document

printf '# Log retention policy v1 (approved 2026-10-04)\n# app logs: 30 days hot, 1 year archive\n# auth/access logs: 90 days hot, 2 years archive\n# audit/security logs: 1 year hot, 7 years archive\n# debug/verbose logs: 7 days hot, no archive\n# deletion: automated by lifecycle rule only, manual deletion requires written approval\n' > /tmp/log-retention-policy.txt && cat /tmp/log-retention-policy.txt

Expected: the policy prints back; get it signed off by whoever owns compliance before implementing.

Step 4: Implement the hot-to-archive lifecycle

aws s3api put-bucket-lifecycle-configuration --bucket my-log-bucket --lifecycle-configuration '{"Rules":[{"ID":"hot-to-archive","Status":"Enabled","Filter":{"Prefix":"logs/"},"Transitions":[{"Days":30,"StorageClass":"GLACIER"}]}]}' && echo "lifecycle applied"

Expected: "lifecycle applied"; logs older than 30 days move to cheap storage automatically, no manual deletion.

Step 5: Prove the policy is actually enforced

aws s3api get-bucket-lifecycle-configuration --bucket my-log-bucket | grep -c hot-to-archive

Expected: count of 1; an auditor wants evidence the rule exists, not just the document.

Variant phrasings

"How long should we keep application logs"

30 days hot for incidents plus a year in cheap archive covers almost every real need; keep security logs longer (years, not months).

"Log retention for SOC 2"

SOC 2 wants a written policy plus evidence it is followed; the lifecycle rule output in step 5 is the evidence.

"Can we delete logs to save money"

Yes, if the deletion follows the written policy and the lifecycle rule does it automatically; manual bulk deletes with no paper trail are what fail audits.

Why it happens

Audits do not fail because retention is 30 days instead of 90; they fail because retention is undocumented, inconsistent, or unenforced. A short written policy plus automated lifecycle rules is the whole game.

Edge cases and pitfalls

• Litigation hold overrides everything; the policy must name who can declare one and how deletions pause.
• Some regulations require immutable storage; check whether archive needs object-lock before you need it.
• Debug logs with PII should have the shortest retention, not the longest; keeping them "just in case" is a liability.
• Review the policy yearly; "we have always done 90 days" is not a rationale an auditor accepts forever.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_CRWpBgrzK2tVz4uB1i3KIA