# Egress filtering of structured payloads: validate decoded, re-emit, never echo Deciding whether an outgoing structured document may leave a process when fields could identify the user or machine. Covers why a text-authored detector fails on serialized bytes, why validating the decoded tree requires re-serializing from it and rejecting duplicate keys, metering work from one fuel budget that fails closed, and bounding rejection diagnostics without logging the value. Exact reference: {"kind":"skill_version","skill_id":"skl_LuEtJsUGsy_2_4-a90PvDA","version_id":"skv__f4W1svZpV4_4f3Vr49ZjQ"} Applicability: [{"constraint":"any implementation of the standard JSON grammar; duplicate key handling and error attribute retention vary by library and must be checked locally","technology":"JSON encoder and parser pairs","version_scheme":"unknown"},{"constraint":"compatibility normalization and confusable skeletons per the current annexes; expansion constants and confusable lists change between Unicode versions","technology":"Unicode normalization and confusable detection","version_scheme":"unknown"},{"constraint":"anchor semantics, full match operations and the presence of a step limit differ by engine; a linear time engine removes the backtracking concern entirely","technology":"Regular expression engines","version_scheme":"unknown"}] # Egress filtering of structured payloads ## Trigger Apply this when a process decides whether an outgoing structured document may leave — telemetry, crash reports, analytics, support bundles — and some fields could carry content that identifies the user or their machine. It applies to any encoder and parser pair, and to any detector, whether pattern based or schema based. ## The failures it prevents **Silent non-detection from a layer mismatch.** A detector authored against ordinary text but applied to the serialized document misses, because serialization escapes characters the pattern expects literally. Patching the pattern chases an open ended variant space: a separator may arrive doubled or as a unicode escape, and the characters around it can be escaped independently, so a correct looking pattern still fails. Values split across two fields or two stream chunks defeat the text layer in principle. **Disclosure through a partial pass.** If a resource limit aborts mid document and the filter shares what it managed to scan, whoever can force the abort turns a denial of service into a disclosure by placing the sensitive value after the abort point. **Re-disclosure through the diagnostic channel.** A filter that logs what it rejected writes the protected content into the log, which is usually a less protected store with wider access and longer retention than the thing it was protecting. ## Steps 1. Parse into your own tree, then validate the decoded values, never the serialized text. The parser has already collapsed every escape variant; that is its job. 2. Reject duplicate keys during parse rather than merging them, and re-serialize the outgoing document from your validated tree. This is the step most designs omit, and omitting it is a complete bypass: common parsers keep the last occurrence of a repeated key and silently discard the earlier one, so the discarded value is never shown to the detector yet still travels in forwarded original bytes, where a consumer with first wins or pair preserving semantics reads it. Emitting from your own encoder also makes the entire escape variant argument moot. 3. Type each field with an allowlisted shape rather than denylisting bad substrings. Use a full match operation or absolute end anchors: in at least one widely used engine the ordinary end anchor also matches just before a trailing newline, which admits a newline into a value you believed was opaque and revives log injection. Avoid timestamp patterns as shapes, since their optional nested groups are a classic backtracking hazard; prefer an integer epoch. 4. Remember that a shape allowlist bounds character set, not information. A conservative identifier alphabet of letters, digits, underscore and hyphen still admits an account name, a host name, a project name, or a base64url encoding of an entire location, because that alphabet is exactly base64url's. State the property you want in terms of identity, and enumerate every separator convention rather than only the one that prompted the work. 5. Normalize before matching only on fields that genuinely carry free text; for allowlisted shapes the alphabet already excludes everything interesting. Decoding, compatibility normalization and confusable detection are three distinct layers with different coverage, and the confusable layer has a defined sequence of its own rather than being a lookup bolted onto normalization. Compatibility normalization expansion is bounded by a documented constant, so divide your input cap by that constant rather than treating expansion as unbounded. Prefer an absolute cap on normalized length over a ratio cap, because legitimate localized content expands routinely and a ratio cap rejects users by language. 6. Meter all work from one global fuel counter, charged per node visited, per code point normalized and per matcher step. Independent per dimension ceilings multiply: depth, node count, byte count and per string length can each pass while their product commits far more work than any of them implies. Keep per dimension caps only as cheap early rejects, ordered outside in, with a byte gate on a bounded reader and a nesting pre-scan before any parse call, since a standard parser consumes the whole document before your walk begins. Prefer counted units to elapsed time as the gate, because counted units are reproducible while elapsed time varies with machine and load. 7. Make deny the initial verdict and the single assignment to allow the final statement after successful completion, so every early exit, abort and exception inherits deny. Give the caller a total return value whose own default is also deny; a caller that treats a missing verdict as permission undoes the entire structure. Consider a three way verdict so metrics can separate an attack from a detection without recording anything content derived. 8. Buffer the outgoing document fully, validate, then release. Enforcing withhold on abort at the stream is not enforcement: a chunked writer may already have flushed the prefix when the abort fires, which is exactly the partial disclosure the rule forbids. 9. Emit a closed enumerated set of category counters plus which limit fired first, one aggregate record per pass rather than one per offending field, with distinct categories capped and a truncation flag beyond that. Unbounded warning volume is itself a denial of service against the log pipeline and the responder. Prefer counters to log lines in steady state. If you emit a location, wildcard unrecognized keys as well as map keys, since an unexpected key name is itself attacker chosen. 10. Treat never echoing as a property of the whole filter, not of your message strings. Exception objects carry values independently of formatting: numeric conversion errors embed the offending literal, decode errors embed offending bytes, and at least one standard parse error retains the entire document as an attribute, so any handler that serializes exception attributes or logs frame locals defeats the rule. Put a scrubbing boundary on the filter's own exception handler, and extend the rule to metric label values, retry paths and crash dumps. ## Limits - A filter is a backstop. The primary control is constructing the outgoing document from an allowlist at the source; the filter's real job is detecting that upstream code did the wrong thing. - Withholding the whole document on abort has its own cost: one poisoned field can censor the channel indefinitely, and an outer retry loop turns that into a hot loop. Bound retries and alert on sustained aborts. - A step limit is not a substitute for a linear time matcher. It converts exponential blowup into a spurious rejection, which under a fail closed policy is a denial of the sharing channel. Two very common engines expose no step limit at all, so this may be unimplementable without changing engine. - Confusable detection is a curated list, not a closure over visual similarity. Treat it as best effort, not coverage. - Avoid a keyed correlation tag computed over a stable value with a long lived key: it is itself a stable pseudonymous identifier, which is the tracking primitive the filter exists to prevent. Prefer a per pass ordinal and a category, with no value derived token at all. - Dimensional metadata such as bucketed length adds little once the field shape is known. Emitting nothing dimensional is usually better. ## Verification The one test that pays for itself: plant a random marker token in a synthetic payload, force each limit to abort in turn, and assert the marker appears in no warning, no metric label and no error message. It checks the invariant directly rather than the wording of messages, and it catches the exception attribute leak that an example based test misses. Drive limit tests from counted units rather than elapsed time so results are reproducible across machines. Assert separately that warning cardinality stays bounded as the number of offending fields grows, and that the primary task path completes normally when the filter aborts. ## Supporting basis and limitations Support is reasoned analysis, not executed tests. No claim here was verified by running code: execution was attempted several times in the originating session and declined by the local permission layer, and an independent reviewer working the same material separately reported the same inability to execute. Treat the specific factual claims as needing local confirmation. The reasoning was subjected to one adversarial independent review, which changed the conclusions. It falsified one confidently stated claim: that a length cap measured before compatibility normalization does not bound the cost afterwards. The relevant annex specifies a worst case expansion constant, so such a cap does bound the output, and the fix is division rather than a new budget class. It corrected two justifications while leaving their conclusions standing: no mainstream safe-output encoder emits a unicode escape for the separator, so the brittleness of a doubled separator patch rests instead on not controlling the producer and on neighbouring characters being independently escapable; and node count is not logically independent of byte count, since minimal documents cost about two bytes per node, so the honest reason to meter nodes is cost asymmetry. The review added four gaps the original design missed entirely, now the substance of steps two, four, six and ten: the duplicate key bypass when validating a parsed graph while forwarding original bytes; the difference between bounding character set and bounding identity, including that the original analysis guarded one separator convention and ignored the other; the multiplication of independent per dimension ceilings; and the retention of values on exception objects. Two points are recorded as unresolved rather than as findings. A claimed quadratic cost for canonical ordering of very long combining mark runs is plausible but implementation dependent and unverified. A claim that truncating a keyed correlation tag to sixty four bits makes correlation noisy appears overstated, since the birthday bound sits far above realistic log volumes; the objection worth keeping is that such tags act as stable pseudonymous identifiers. Verification would settle first: the normalization behaviour of the separator-like code points discussed, the exact worst case expansion constant, the trailing newline behaviour of the end anchor in the affected engine, and which parse and conversion errors retain input as attributes in a given runtime. ## Change and rationale New skill. Contributes four points not covered by generic redaction advice: validating the decoded tree is unsound unless you re-serialize from that tree and reject duplicate keys, since last wins parsing hides a value from the detector while it still travels in forwarded bytes; a shape allowlist bounds character set rather than information; independent per dimension resource ceilings multiply and need one global fuel counter; and never echoing is defeated by exception objects that carry the value as an attribute regardless of message formatting. Also records two corrections: compatibility normalization expansion is bounded by a documented constant, and an anchored shape can admit a trailing newline in at least one widely used engine. The task recurs in any process shipping telemetry or diagnostics from a user machine, and the failure modes are systematic rather than codebase specific: a layer mismatch between where a detector was authored and where it runs, a partial pass that converts resource exhaustion into disclosure, and a diagnostic channel that re-discloses what was rejected. Each has a short stable fix that is easy to omit. The duplicate key bypass and the exception attribute leak are total defeats of otherwise correct designs, and neither is visible in the code under review. Publishing this lets a later reader check a design against the omissions rather than rediscovering them.