agent leaked a session token into its own logs: detect and rotate

agent leaked a session token into its own logs: detect and rotate

TL;DR

Treat it as compromised until proven otherwise. Rotate the credential now, find every place the log went, scrub it, then add redaction so it can't happen again.

agent leaked a session token into its own logs: detect and rotate

Use this when

• A session token, API credential, or similar secret appears in agent log output
• The value shows up in a log aggregator, error tracker, or chat alert
• A code review or audit spots credential-shaped strings in logs
• You need the incident playbook, not just the rotation command

Not for this skill when

• The credential was committed to git (rotate, then purge history; logs are the easy case)
• You're doing routine rotation with no leak
• The "leak" is a test credential with no access

Steps

1. Rotate immediately. Revoke the leaked credential at the issuer and issue a new one. Don't wait to assess blast radius first.

Expected: the old value stops working everywhere within minutes.

2. Find the exposure. Search the log aggregator for the value's distinctive prefix or pattern and list every sink that received it: files, central logging, error trackers, chat alerts.

grep -rl "sess_" /var/log/agent/ | head -20

Expected: a concrete list of files and sinks to scrub. (Use the real prefix your credentials carry.)

3. Scrub each sink. For local files:

grep -rl "sess_" /var/log/agent/ | xargs sed -i 's/sess_[A-Za-z0-9]*/[redacted]/g'

For hosted sinks, use their deletion or redaction feature. Then re-search to confirm. Expected: re-searching finds no hits.

4. Add a redaction filter at the agent's logging layer: scrub authorization headers, known credential shapes, and your credential prefixes before anything hits a sink. Test it with a fake value.

Expected: a test log line containing a fake credential comes out redacted.

5. Write the incident note: what leaked, where it went, when rotation completed, what redaction was added. If the credential touched customer data or a regulated scope, loop in whoever owns breach assessment.

Expected: a short written record, not just a memory.

Variant: credential leaked into a git commit

Rotate, then purge with filter-repo or BFG. The commit is forever otherwise; scrubbing the working tree isn't enough.

Variant: leaked into a chat channel or ticket

Rotate, delete the message, and check channel history exports and notification emails that may have captured it.

Variant: leaked via error tracker breadcrumbs

Scrub the issue in the tracker and add the redaction to the SDK's before-send hook so breadcrumbs are cleaned at the source.

Why this happens

Logs are the most replicated data you have: forwarders, archives, backups, search indexes. A credential in a log line is effectively published to everyone with log access, and retention means it stays valid-looking for months. Rotation is the only reliable containment; scrubbing is hygiene.

Edge cases and pitfalls

• Backups may still hold the old logs; note the retention window in the incident record.
• Don't pass the real credential around while coordinating; reference it by name and a few trailing characters.
• If the same value was used in multiple environments, rotate all of them; agents reuse credentials more than people admit.
• Redaction regexes need tests; an over-broad pattern redacts useful debugging, an under-broad one misses the next leak.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_3VXHPsDMNA9yMkeSXr4AZg