Egress filtering of structured payloads

Trigger

Apply this when a process decides whether an outgoing structured document may leave — telemetry, crash reports, analytics, support bundles — and some fields could carry content that identifies the user or their machine. It applies to any encoder and parser pair, and to any detector, whether pattern based or schema based.

The failures it prevents

Silent non-detection from a layer mismatch. A detector authored against ordinary text but applied to the serialized document misses, because serialization escapes characters the pattern expects literally. Patching the pattern chases an open ended variant space: a separator may arrive doubled or as a unicode escape, and the characters around it can be escaped independently, so a correct looking pattern still fails. Values split across two fields or two stream chunks defeat the text layer in principle.

Disclosure through a partial pass. If a resource limit aborts mid document and the filter shares what it managed to scan, whoever can force the abort turns a denial of service into a disclosure by placing the sensitive value after the abort point.

Re-disclosure through the diagnostic channel. A filter that logs what it rejected writes the protected content into the log, which is usually a less protected store with wider access and longer retention than the thing it was protecting.

Steps

1. Parse into your own tree, then validate the decoded values, never the serialized text. The parser has already collapsed every escape variant; that is its job.

1. Reject duplicate keys during parse rather than merging them, and re-serialize the outgoing document from your validated tree. This is the step most designs omit, and omitting it is a complete bypass: common parsers keep the last occurrence of a repeated key and silently discard the earlier one, so the discarded value is never shown to the detector yet still travels in forwarded original bytes, where a consumer with first wins or pair preserving semantics reads it. Emitting from your own encoder also makes the entire escape variant argument moot.

1. Type each field with an allowlisted shape rather than denylisting bad substrings. Use a full match operation or absolute end anchors: in at least one widely used engine the ordinary end anchor also matches just before a trailing newline, which admits a newline into a value you believed was opaque and revives log injection. Avoid timestamp patterns as shapes, since their optional nested groups are a classic backtracking hazard; prefer an integer epoch.

1. Remember that a shape allowlist bounds character set, not information. A conservative identifier alphabet of letters, digits, underscore and hyphen still admits an account name, a host name, a project name, or a base64url encoding of an entire location, because that alphabet is exactly base64url's. State the property you want in terms of identity, and enumerate every separator convention rather than only the one that prompted the work.

1. Normalize before matching only on fields that genuinely carry free text; for allowlisted shapes the alphabet already excludes everything interesting. Decoding, compatibility normalization and confusable detection are three distinct layers with different coverage, and the confusable layer has a defined sequence of its own rather than being a lookup bolted onto normalization. Compatibility normalization expansion is bounded by a documented constant, so divide your input cap by that constant rather than treating expansion as unbounded. Prefer an absolute cap on normalized length over a ratio cap, because legitimate localized content expands routinely and a ratio cap rejects users by language.

1. Meter all work from one global fuel counter, charged per node visited, per code point normalized and per matcher step. Independent per dimension ceilings multiply: depth, node count, byte count and per string length can each pass while their product commits far more work than any of them implies. Keep per dimension caps only as cheap early rejects, ordered outside in, with a byte gate on a bounded reader and a nesting pre-scan before any parse call, since a standard parser consumes the whole document before your walk begins. Prefer counted units to elapsed time as the gate, because counted units are reproducible while elapsed time varies with machine and load.

1. Make deny the initial verdict and the single assignment to allow the final statement after successful completion, so every early exit, abort and exception inherits deny. Give the caller a total return value whose own default is also deny; a caller that treats a missing verdict as permission undoes the entire structure. Consider a three way verdict so metrics can separate an attack from a detection without recording anything content derived.

1. Buffer the outgoing document fully, validate, then release. Enforcing withhold on abort at the stream is not enforcement: a chunked writer may already have flushed the prefix when the abort fires, which is exactly the partial disclosure the rule forbids.

1. Emit a closed enumerated set of category counters plus which limit fired first, one aggregate record per pass rather than one per offending field, with distinct categories capped and a truncation flag beyond that. Unbounded warning volume is itself a denial of service against the log pipeline and the responder. Prefer counters to log lines in steady state. If you emit a location, wildcard unrecognized keys as well as map keys, since an unexpected key name is itself attacker chosen.

1. Treat never echoing as a property of the whole filter, not of your message strings. Exception objects carry values independently of formatting: numeric conversion errors embed the offending literal, decode errors embed offending bytes, and at least one standard parse error retains the entire document as an attribute, so any handler that serializes exception attributes or logs frame locals defeats the rule. Put a scrubbing boundary on the filter's own exception handler, and extend the rule to metric label values, retry paths and crash dumps.

Limits

• A filter is a backstop. The primary control is constructing the outgoing document from an allowlist at the source; the filter's real job is detecting that upstream code did the wrong thing.
• Withholding the whole document on abort has its own cost: one poisoned field can censor the channel indefinitely, and an outer retry loop turns that into a hot loop. Bound retries and alert on sustained aborts.
• A step limit is not a substitute for a linear time matcher. It converts exponential blowup into a spurious rejection, which under a fail closed policy is a denial of the sharing channel. Two very common engines expose no step limit at all, so this may be unimplementable without changing engine.
• Confusable detection is a curated list, not a closure over visual similarity. Treat it as best effort, not coverage.
• Avoid a keyed correlation tag computed over a stable value with a long lived key: it is itself a stable pseudonymous identifier, which is the tracking primitive the filter exists to prevent. Prefer a per pass ordinal and a category, with no value derived token at all.
• Dimensional metadata such as bucketed length adds little once the field shape is known. Emitting nothing dimensional is usually better.

Verification

The one test that pays for itself: plant a random marker token in a synthetic payload, force each limit to abort in turn, and assert the marker appears in no warning, no metric label and no error message. It checks the invariant directly rather than the wording of messages, and it catches the exception attribute leak that an example based test misses.

Drive limit tests from counted units rather than elapsed time so results are reproducible across machines. Assert separately that warning cardinality stays bounded as the number of offending fields grows, and that the primary task path completes normally when the filter aborts.