# Egress JSON redaction: validate decoded values, budget the work, never echo rejections Use when a filter must stop identifying strings from leaving in outgoing JSON and must survive hostile depth, size and matcher cost. Covers matching at every decode round, allowlists versus matcher cost, a proportional budget ledger, why a raw pre-scan and an abandoned worker are not controls, two senses of failing closed, and rejection telemetry that cannot leak the value. Exact reference: {"kind":"skill_version","skill_id":"skl_uU6sh4MeLBJCsV31NKsFAA","version_id":"skv_cLRsMZ660LOGZbemO5AmlQ"} Applicability: [{"constraint":"Any language or runtime; assumes a parsed tree and a canonical serializer are available","technology":"Structured document egress filtering","version_scheme":"unknown"},{"constraint":"Compatibility normalization does not fold confusables; a separate confusable skeleton stage is assumed","technology":"Unicode normalization and confusable detection","version_scheme":"unknown"}] # Egress JSON redaction: validate decoded values, budget the work, never echo rejections ## When this applies A component inspects structured documents before an optional outbound path transmits them, and must prevent identifying strings, typically filesystem paths, from leaving. Reach for this whenever a redaction rule is written as a pattern, and whenever such a filter is exposed to input an attacker can shape. ## The failure it prevents A pattern authored against ordinary readable text, then applied to the serialized document, silently never matches, because the separator a human reads as one character is escaped into two on the wire. A rule of the form letter, colon, separator, then one non-separator consumes the first backslash of the pair and fails on the second, and no later offset begins with a letter and colon, so no backslash-separated path of any shape matches. The same class covers unicode escapes, percent encoding, base64, and a document nested inside a string value. Patching the pattern to expect the escaped form closes one case and leaves the class open. Three further designs look like controls and are not: matching only after a normalization pass, a raw-byte bracket pre-scan for depth, and an abandon-on-deadline worker thread. Each is addressed below. ## Steps **1. Validate the decoded value at every decode round, and return the bytes you validated.** Parse, walk the tree, and apply every rule to each decoded string at round zero, meaning the string as the JSON parser produced it, and again after each further decode round. A match at any round rejects. Decode rounds are strict: the four-hex-digit unicode escape form only, percent decoding, and nested-document parsing. Never use a general backslash unescaper, because it turns path segments that begin with t, b, n, r or f into control characters and the path stops looking like a path. If a nested string fails strict parsing, keep it as opaque text and match on that; do not reject the outer document for it. Serialize canonically from the round-zero tree. The allow result must carry those canonical bytes, and the outbound path must be structurally unable to transmit anything else. A boolean verdict reopens the bug, since the caller then reserializes with a different encoder. Approve the complete outbound body: an envelope the caller wraps around approved bytes is unfiltered. **2. Allowlist field shapes, but infer nothing about matcher cost from that.** Declare a permitted shape per field and reject unknown fields by default, because denylisting identifying shapes is unbounded. A narrow allowlist shape is exactly where nested quantifiers appear. Require a linear-time engine, or statically verify each pattern is backtracking-free, and keep a matcher step budget as defense in depth. A hard length cap alone does not tame an exponential pattern. Where free text is unavoidable, separate identifying-path signals, such as a user-directory segment or a drive letter followed by a user segment, from merely path-shaped signals such as a leading separator, a doubled separator, or a letter before a colon. The path-shaped signals fire on URL schemes, HTTP routes, dates, and every system frame in a stack trace, which makes crash reports unshareable. Reject free text only on identifying signals, and anchor drive-letter signals at token start. **3. Reject invisible characters rather than normalizing them away.** Compatibility normalization does not remove zero-width spaces, soft hyphens, joiners or bidirectional controls, and one of them inside a prefix defeats an anchored signal while rendering identically. Reject format and bidirectional controls in constrained fields. Compatibility normalization folds fullwidth colons and slashes but not confusables such as set minus, division slash, or Cyrillic lookalike letters; use a separate confusable skeleton or do not claim that coverage. **4. Score identifying shape per token, not per string.** Signals anchored at string start or end, plus a whole-string separator density, are bypassed by embedding the value mid-sentence in any free prose field. Split the decoded value on whitespace, compute every signal per token, and reject the document if any token scores. Never rewrite a scoring value; rewriting is how partial identifiers survive. **5. Route every cost through one budget ledger, charged before the work and proportional to it.** Cap raw bytes before parsing, container depth, node count, keys per object, per-string decoded length, an aggregate string total, decode rounds, and normalized output length charged cumulatively per document, since compatibility normalization can expand one code point into many. Charge before each unit, and make each charge bytes processed times a per-operation constant. A flat charge per decode round, per diagnostic hash or per matcher call lets one call do a megabyte of uncharged work. Enforce depth inside the parser with an explicit counter and pass the same ledger to the nested-document walker. Do not use a raw-byte bracket pre-scan: it counts brackets inside string literals and falsely rejects legitimate values, a string-aware version is a second tokenizer that creates a parser differential, and neither can see depth contributed by nested decoding. Carry numbers as validated raw text so canonical serialization cannot silently alter large integers. Reject lone surrogate escapes at parse time, and compare duplicate keys after escape decoding. Make parse, decode, walk and serialize non-recursive. **6. Fail closed in two distinct senses.** Initialize the verdict to deny and assign allow only as the final statement after a completed clean walk. Separately, make the filter total toward its caller: a boundary catch-all maps any escape to deny rather than propagating, so the optional outbound path is suppressed while the primary work continues. On any abort withhold the whole document, never the scanned portion, because burying a value past the abort point converts a denial of service into a disclosure. An abandoned worker thread protects the caller's latency only. In most runtimes it keeps consuming CPU and memory, and an out-of-memory failure reaches the caller. Isolation that counts is a killable process or isolate with memory and CPU-time limits, plus a circuit breaker that denies sharing after repeated abandonments in a window. Stack overflow is not reliably catchable, so do not depend on catching it. **7. Make rejection telemetry incapable of carrying the value.** Log a field pointer derived from the schema, a rule identifier set from a closed enumeration, the decode round, a coarse length bucket, and the budget name with its configured limit. Use wildcard locations for arrays and maps, since an index is data-derived, and never emit an unknown field name. Drop character-class summaries: combined with location and length they profile a small value space, and a non-ASCII flag is a locale signal. Emit one record per value, not one per rule. A keyed hash for correlating false positives must use a per-installation key held outside the log store, since a co-located key makes short values brute-forceable and a fleet-wide key makes the hash a cross-user pseudonym. Rotation contradicts correlation, so choose the window deliberately, include a window identifier, and treat the hash as local-debug-only. Index counters by the closed enumeration, translate engine errors into local codes because parser and matcher messages embed subject fragments, and rate-limit and deduplicate per category per window. **8. Test the invariants, through the wire path, without touching the environment.** Embed each generated candidate in a document, serialize it, and call the public entry point on the bytes, with a mock transmitter asserting it never received bytes containing the candidate. A property test that only feeds decoded strings to the detector never exercises the layer where the original defect lived. Enumerate escape variants: doubled backslash, both hex cases of the unicode escape, escaped and unicode forward slash, and mixtures within one value. Pair each identifying case with a benign control that must be allowed, or a filter that denies everything passes. Assert on rendered log output, stderr and panic text, not on the record type. Generate candidates from a placeholder grammar that never reads the environment, or a shrunk counterexample prints a real path in CI logs. Prove early abort by asserting on consumed budget units rather than elapsed time. Include a false-rejection test: a string value containing thousands of literal open brackets must be allowed. ## Limits Counted budgets are deterministic, but a wall-clock backstop makes the verdict deterministic only modulo that deadline; run determinism tests with a virtual or disabled clock. Derive caps from a high percentile of legitimate payload shape under a compiled ceiling that configuration cannot raise; a byte-exceeded counter is the signal that legitimate growth is hitting the cap. State whether the adversary is an accidental encoding layer or a hostile producer, since against a hostile producer no denylist wins and that decision bounds how many decode rounds are worth their false-positive cost. Paths split across fields or array elements are invisible to per-string rules; only schema allowlists cover them. Whether naming the budget that aborted first discloses useful structure to a probing attacker remains open. ## Basis Reasoned analysis and two independent adversarial reviews. Nothing here was measured or executed; every threshold is a placeholder the adopting team must derive and test. ## Supporting basis and limitations Support is reasoned analysis only. No tests were executed, no code was run, no timings were measured, and no repository or configuration was inspected. The design was drafted, extended with resource budgets, and then checked by one independent reviewer that also executed nothing. Derivations the review confirmed: the character-by-character account of why the letter, colon, separator, non-separator rule fails on a doubled backslash; validating decoded values and emitting the validated bytes; walking keys and array elements; rejecting the whole document on abort; one per-document ledger; backtracking as an engine property; two separate senses of failing closed; fixed limits; closed-enum warnings with counters; compatibility normalization expanding output; a keyed function rather than a plain digest for short values. Concrete errors the review found and this version corrects: running rules only after normalization, where a general unescaper mangles path segments beginning with t, b, n, r or f and the untouched original is what ships; an abandoned worker that bounds latency but not CPU or memory, so out-of-memory reaches the caller; a raw-byte bracket pre-scan that counts brackets inside string literals, creates a tokenizer differential, and cannot see nested-decode depth; canonical serialization that can alter large integers and cannot re-emit lone surrogates, plus an unfiltered caller envelope; a rotating hash key that defeats the correlation it exists for, a fleet-wide key that becomes a cross-user pseudonym, and a co-located key that enables brute force; unanchored path-shaped signals that fire on URL schemes and every frame of a stack trace; flat budget charges per decode round, hash or matcher call; a length cap presented as sufficient against an exponential pattern; property tests that never traverse serialization; strict nested parsing that rejected the outer document; a rejection record whose character-class summary and array indexes profile small value spaces; and a claim that compatibility normalization folds confusables. Remaining unknowns: a default normalization expansion cap, whether naming the aborting budget leaks structure, and whether process isolation is worth its cost when the matcher engine cannot be replaced. ## Change and rationale Folds in corrections from a second adversarial pass. Rules now run at round zero and after every decode round, with strict four-hex-digit unicode decoding, because a general unescaper hides paths whose segments begin with t, b, n, r or f. Removes the raw-byte bracket pre-scan in favor of parser-enforced depth on the shared ledger. Makes budget charges proportional to bytes. Replaces the abandon-on-deadline worker with a killable isolate plus circuit breaker. Separates identifying-path signals from path-shaped signals to keep crash reports shareable. Fixes the keyed-hash guidance so rotation does not defeat correlation. Requires tests to go through serialization and to never read the environment. The prior version was sound on layering, ledgers and telemetry but still contained four items that look like controls and are not: matching only after normalization, a raw bracket pre-scan, flat per-call budget charges, and an abandoned worker thread. Each was asserted confidently in the originating discussion and then falsified by an independent review, which is the signal that they are easy to get wrong. Updating the existing skill rather than creating a sibling keeps one canonical body of guidance for this exact topic and preserves the earlier falsified claims in the limits.