When this applies

Use this when a service screens outgoing structured payloads for string values that must not cross a boundary, such as filesystem paths, account names or internal identifiers, and the screening is implemented as pattern matching. The trigger is any filter whose rules were authored against ordinary text but run against serialized output, or any filter that must keep the rejected value out of its own logs.

The failure it prevents

A rule authored for decoded text and applied to serialized bytes fails in both directions.

Missed detection. In a JSON string every literal backslash is emitted as a pair. A rule shaped like "drive letter, colon, separator, then one or more non separator characters" consumes the first backslash of the pair, then requires a non separator, but the next character is the second backslash. It fails. This generalizes: the consumed separator is always the first of a pair, since being the second would require the preceding colon to be a separator. So no backslash separated path of any shape matches that rule in escaped form.

False detection, which is the failure more likely to page someone. The same escaping supplies separators that were never in the value. Benign prose containing a letter, a colon, and then any escaped control character or quote matches the same rule.

A correction worth recording, because the wrong version is plausible. Numeric character escapes are not an evasion of a byte level scan when the escape is introduced by the separator itself: such an escape begins with the very character it encodes, so the character is present in the raw bytes and the naive scan does fire. The evasion is real only for encodings whose escape does not contain the separator, such as percent encoding, markup entities, base64 and quoted printable.

Steps

1. Evaluate decoded values. Parse with a strict parser and apply rules to decoded scalars, or apply them before serialization. Deny unparseable or non canonical input, otherwise the evasion is simply a body the filter's parser cannot read. Watch for parser differentials against the receiver: duplicate keys, trailing data, byte order marks, comments, non finite numbers, invalid UTF-8, lone surrogates.

1. Constrain each field with a declared grammar before any heuristic runs. Enumerations, identifiers, bounded token charsets. This removes whole evasion families rather than detecting them. Free text fields get no grammar and must always run the heuristic; a grammar admitting any string up to some length is not a control.

1. Order canonicalization correctly: remove format and zero width characters, compatibility normalize, full case fold, normalize again, then unify separators. The combined normalization and case folding form performs the middle steps correctly on its own. Two reasons the order matters. Format characters carry combining class zero and block canonical composition, so stripping after normalizing leaves output that is not normalized. And full case folding is not closed under normalization, so normalizing then folding can oscillate with period two and never converge, which combines badly with a rule that rejects on non convergence. Separator unification stays last because compatibility normalization creates separators.

1. Decode nested encodings only where the declared grammar says an encoding is expected. Speculative decoding of every string leaf is a false positive generator: scanning arbitrary decoded binary with a short structural predicate yields on the order of a few spurious matches per megabyte, so any payload carrying an image or attachment denies almost always. Exclude decompression unless separately budgeted, since it is the one step with an unbounded expansion ratio.

1. Bound the work. Cap encoded bytes before allocating. Enforce depth during parsing rather than after, because the catch site for stack exhaustion is nondeterministic and leaves a live copy of the value in arbitrary frames. Cap node count, element count and string length separately, since depth and size are independent axes. Carry one budget through the whole traversal instead of granting a fresh allowance per nested call. Charge canonicalization after expansion. Prefer a deterministic step counter over a wall clock deadline, and traverse keys in sorted order, or the counter's reproducibility is lost to unordered map iteration.

1. Prefer a pattern engine with linear time guarantees. Path shape predicates need neither backreferences nor lookaround. Set the engine's pattern size or memory limit explicitly, since such engines are linear in input but can still blow up during pattern compilation. Do not rely on a lint for nested quantifiers: ambiguity under a single quantifier, such as an alternation of two identical branches, is exponential with no nesting present.

1. Fail closed for the sharing decision, open for the primary work. Return allow, deny or undetermined, and treat undetermined as deny. Require allow to be an explicit positive token compared by equality, never a test for "not deny", since only some languages have meaningful type defaults. Catch filter exceptions at the boundary and map them to deny, excluding cancellation and shutdown signals. Alert on the deny rate: a completely broken filter denies everything and looks maximally secure.

1. Consider replacing an offending value with a fixed width constant token rather than rejecting the message. Partial redaction is unsafe because the segment you keep is often the identifying one, but whole value replacement leaks nothing and preserves availability.

1. Keep rejection records content free. Record the declared schema position rather than data derived keys, a rule identifier, coarse fixed buckets rather than exact measurements, boolean class flags, and a keyed digest for correlation. Keyed, not a plain hash: the candidate space is small enough to enumerate offline. Treat the key's lifetime as the correlation window, and note that crash dump capture is a key compromise channel. Do not log per character shape templates; their information content grows with value length, and they are content rather than metadata. Rate limit and deduplicate the warnings themselves, keyed on rule and position, never on the value.

1. Test with an adversarial corpus and two properties. Property one: embed a distinctive marker in the payload and assert it never appears in logs, metric labels, trace attributes, returned error text or core dumps, including its encoded variants, or the test passes vacuously. Property two, stated one sided: if the raw value denies, no encoding of it may allow. One sided and under a disabled budget, because a shared budget makes the decision depend on the rest of the message and a two sided round trip equality will flake and then be switched off.

Limits

• A shared budget deliberately makes the decision non local. Accept that and test budget accounting separately rather than asserting value level determinism.
• Per leaf evaluation does not catch a value split across two fields that the receiver concatenates. Whole message reconstruction is a separate control.
• Compatibility normalization does not fold confusable letters from other scripts, so a lookalike drive letter survives it. Add a confusable skeleton mapping or a script mixing check if that is in scope.
• The allow or deny decision is itself an oracle. Where message content is influenced by an untrusted party, a visible deny leaks one bit per probe, undermining the effort spent keeping bytes out of logs. Fixed token replacement, generic failures at the trust boundary, or per caller rate limiting are the mitigations.
• Bounding the filter does not bound the rest of the pipeline processing the same hostile input.

Evidence status

Reasoned analysis and adversarial review only. No tests were executed. Verify the character data claims against the current character database, and the runtime behaviour claims against the target runtime, before relying on them.