Egress JSON redaction: validate decoded values, budget the work, never echo rejections

When this applies

A component inspects structured documents before an optional outbound path transmits them, and must prevent identifying strings, typically filesystem paths, from leaving. Reach for this whenever a redaction rule is written as a pattern, and whenever such a filter is exposed to input an attacker can shape.

The failure it prevents

A pattern authored against ordinary readable text, then applied to the serialized document, silently never matches, because the separator a human reads as one character is escaped into two on the wire. A rule of the form letter, colon, separator, then one non-separator consumes the first backslash of the pair and fails on the second, and no later offset begins with a letter and colon, so no backslash-separated path of any shape matches. The same class covers unicode escapes, percent encoding, base64, and a document nested inside a string value. Patching the pattern to expect the escaped form closes one case and leaves the class open.

Three further designs look like controls and are not: matching only after a normalization pass, a raw-byte bracket pre-scan for depth, and an abandon-on-deadline worker thread. Each is addressed below.

Steps

1. Validate the decoded value at every decode round, and return the bytes you validated. Parse, walk the tree, and apply every rule to each decoded string at round zero, meaning the string as the JSON parser produced it, and again after each further decode round. A match at any round rejects. Decode rounds are strict: the four-hex-digit unicode escape form only, percent decoding, and nested-document parsing. Never use a general backslash unescaper, because it turns path segments that begin with t, b, n, r or f into control characters and the path stops looking like a path. If a nested string fails strict parsing, keep it as opaque text and match on that; do not reject the outer document for it. Serialize canonically from the round-zero tree. The allow result must carry those canonical bytes, and the outbound path must be structurally unable to transmit anything else. A boolean verdict reopens the bug, since the caller then reserializes with a different encoder. Approve the complete outbound body: an envelope the caller wraps around approved bytes is unfiltered.

2. Allowlist field shapes, but infer nothing about matcher cost from that. Declare a permitted shape per field and reject unknown fields by default, because denylisting identifying shapes is unbounded. A narrow allowlist shape is exactly where nested quantifiers appear. Require a linear-time engine, or statically verify each pattern is backtracking-free, and keep a matcher step budget as defense in depth. A hard length cap alone does not tame an exponential pattern. Where free text is unavoidable, separate identifying-path signals, such as a user-directory segment or a drive letter followed by a user segment, from merely path-shaped signals such as a leading separator, a doubled separator, or a letter before a colon. The path-shaped signals fire on URL schemes, HTTP routes, dates, and every system frame in a stack trace, which makes crash reports unshareable. Reject free text only on identifying signals, and anchor drive-letter signals at token start.

3. Reject invisible characters rather than normalizing them away. Compatibility normalization does not remove zero-width spaces, soft hyphens, joiners or bidirectional controls, and one of them inside a prefix defeats an anchored signal while rendering identically. Reject format and bidirectional controls in constrained fields. Compatibility normalization folds fullwidth colons and slashes but not confusables such as set minus, division slash, or Cyrillic lookalike letters; use a separate confusable skeleton or do not claim that coverage.

4. Score identifying shape per token, not per string. Signals anchored at string start or end, plus a whole-string separator density, are bypassed by embedding the value mid-sentence in any free prose field. Split the decoded value on whitespace, compute every signal per token, and reject the document if any token scores. Never rewrite a scoring value; rewriting is how partial identifiers survive.

5. Route every cost through one budget ledger, charged before the work and proportional to it. Cap raw bytes before parsing, container depth, node count, keys per object, per-string decoded length, an aggregate string total, decode rounds, and normalized output length charged cumulatively per document, since compatibility normalization can expand one code point into many. Charge before each unit, and make each charge bytes processed times a per-operation constant. A flat charge per decode round, per diagnostic hash or per matcher call lets one call do a megabyte of uncharged work. Enforce depth inside the parser with an explicit counter and pass the same ledger to the nested-document walker. Do not use a raw-byte bracket pre-scan: it counts brackets inside string literals and falsely rejects legitimate values, a string-aware version is a second tokenizer that creates a parser differential, and neither can see depth contributed by nested decoding. Carry numbers as validated raw text so canonical serialization cannot silently alter large integers. Reject lone surrogate escapes at parse time, and compare duplicate keys after escape decoding. Make parse, decode, walk and serialize non-recursive.

6. Fail closed in two distinct senses. Initialize the verdict to deny and assign allow only as the final statement after a completed clean walk. Separately, make the filter total toward its caller: a boundary catch-all maps any escape to deny rather than propagating, so the optional outbound path is suppressed while the primary work continues. On any abort withhold the whole document, never the scanned portion, because burying a value past the abort point converts a denial of service into a disclosure. An abandoned worker thread protects the caller's latency only. In most runtimes it keeps consuming CPU and memory, and an out-of-memory failure reaches the caller. Isolation that counts is a killable process or isolate with memory and CPU-time limits, plus a circuit breaker that denies sharing after repeated abandonments in a window. Stack overflow is not reliably catchable, so do not depend on catching it.

7. Make rejection telemetry incapable of carrying the value. Log a field pointer derived from the schema, a rule identifier set from a closed enumeration, the decode round, a coarse length bucket, and the budget name with its configured limit. Use wildcard locations for arrays and maps, since an index is data-derived, and never emit an unknown field name. Drop character-class summaries: combined with location and length they profile a small value space, and a non-ASCII flag is a locale signal. Emit one record per value, not one per rule. A keyed hash for correlating false positives must use a per-installation key held outside the log store, since a co-located key makes short values brute-forceable and a fleet-wide key makes the hash a cross-user pseudonym. Rotation contradicts correlation, so choose the window deliberately, include a window identifier, and treat the hash as local-debug-only. Index counters by the closed enumeration, translate engine errors into local codes because parser and matcher messages embed subject fragments, and rate-limit and deduplicate per category per window.

8. Test the invariants, through the wire path, without touching the environment. Embed each generated candidate in a document, serialize it, and call the public entry point on the bytes, with a mock transmitter asserting it never received bytes containing the candidate. A property test that only feeds decoded strings to the detector never exercises the layer where the original defect lived. Enumerate escape variants: doubled backslash, both hex cases of the unicode escape, escaped and unicode forward slash, and mixtures within one value. Pair each identifying case with a benign control that must be allowed, or a filter that denies everything passes. Assert on rendered log output, stderr and panic text, not on the record type. Generate candidates from a placeholder grammar that never reads the environment, or a shrunk counterexample prints a real path in CI logs. Prove early abort by asserting on consumed budget units rather than elapsed time. Include a false-rejection test: a string value containing thousands of literal open brackets must be allowed.

Limits

Counted budgets are deterministic, but a wall-clock backstop makes the verdict deterministic only modulo that deadline; run determinism tests with a virtual or disabled clock. Derive caps from a high percentile of legitimate payload shape under a compiled ceiling that configuration cannot raise; a byte-exceeded counter is the signal that legitimate growth is hitting the cap. State whether the adversary is an accidental encoding layer or a hostile producer, since against a hostile producer no denylist wins and that decision bounds how many decode rounds are worth their false-positive cost. Paths split across fields or array elements are invisible to per-string rules; only schema allowlists cover them. Whether naming the budget that aborted first discloses useful structure to a probing attacker remains open.

Basis

Reasoned analysis and two independent adversarial reviews. Nothing here was measured or executed; every threshold is a placeholder the adopting team must derive and test.