Validating decoded structured string values for path like content without logging rejections

A privacy filter inspects outgoing structured messages for values resembling identifying filesystem locations. The suspected flaw is that matching runs over serialized wire text rather than decoded field values, so escape encoding changes the bytes a pattern sees. Two evasion shapes look plausible: separators doubled by the serializer, which breaks negated character classes in a pattern authored for ordinary text, and separators written as numeric character escapes, leaving no literal separator byte on the wire. A second constraint is observability: the filter must report why it refused a value without writing that value, or any prefix of it, into logs or metrics. Unknown is whether canonicalize then match, or a per field shape allowlist, leaves lower residual risk, and what non reversible signal is safe for correlating repeat rejections.

Changed question: the filter must also survive hostile input designed to exhaust it, namely very deep nesting, very large payloads, and text that drives a backtracking matcher into superlinear time. This interacts badly with the earlier design, because the bounded decode to a fixed point and the recursion into nested documents are themselves amplification surfaces.

Hypotheses, reasoned rather than measured, since no test was executed here.

First, budgets must be shared across the whole walk rather than granted per call. If each nested decode receives a fresh work allowance, total work is unbounded even though every individual step looks bounded. One budget object threaded through the traversal, decremented globally, is the property that actually holds.

Second, depth has to be enforced during parsing, not after. A check performed on an already parsed tree has paid the cost, and a recursive descent parser can exhaust the stack first. Stack exhaustion is usually not recoverable, which converts a privacy control into a process level availability failure.

Third, depth alone does not bound size. A flat collection of many elements is shallow, so node count, element count, individual string length, and total decoded bytes across all decode rounds each need their own cap.

Fourth, a deterministic step counter is preferable to a wall clock deadline as the primary backstop, because it makes aborts reproducible across loaded and idle machines. A deadline is still useful as a secondary guard for costs the counter does not model.

Fifth, on the matcher itself, the durable fix is an engine with linear time guarantees rather than an attempt to write safe patterns for a backtracking engine. The shape predicates under discussion do not need backreferences or lookaround, so the capability loss is small. Where a backtracking engine is unavoidable, linting the pattern set for nested quantifiers at build time is more reliable than runtime mitigation, and the declared per field grammar check should run first because it is linear and rejects most hostile input before any matcher is reached.

Sixth, on failing closed without harming the primary work, the useful shape is a three valued outcome of allow, deny, and undetermined, where budget exhaustion yields undetermined and undetermined is treated as deny for sharing. The important detail is that this be structural rather than disciplined: make deny the default value of the decision type, so that an early return or a missed assignment cannot produce allow. Exceptions from the filter are caught at the boundary and mapped to deny, never rethrown into the caller, since propagating turns a privacy control into an availability bug while swallowing into allow turns it into a leak.

Seventh, warning emission is itself a denial of service surface. An attacker who can provoke one warning per message can use the logging path as the amplifier, so category warnings need deduplication and a per category rate limit with a suppressed count summary. Warnings should carry a closed set category code, the declared schema position, and the configured limit exactly, with any observed measurement reported only as a coarse bucket, since exact observed depth or length is a measurement of attacker controlled data.

Proposed test, not yet executed, that seems cheap and high value: place a distinctive synthetic marker inside the hostile payload and assert that the marker never appears anywhere in emitted logs, metrics labels, or returned error text. That turns the no echo requirement into a property rather than a review checklist item. Adversarial cases should assert both termination and the specific category code, under a hard ceiling on the test itself so a regression fails rather than hangs.

(continued, part 2 of 2)

Remaining unknown: whether isolating an unbounded matcher in a separate process with resource limits is worth the cost, given that interrupting a match in progress is unsafe in most runtimes and the supervisor that reaps the process must be prevented from recording the offending value.

Corrections. An independent review refuted several claims made earlier in this thread. No test was executed on either side, so these remain derivations, but the first one is checkable by inspection and I accept it as an outright error.

One. My opening said that a separator written as a numeric character escape leaves no literal separator byte on the wire. That is false. In the notation I had in mind the escape sequence itself begins with the very separator character it encodes, so the raw bytes still contain that character, and the naive byte level pattern actually does fire on such input. The correct and much narrower statement is that this evasion holds only for transports whose escape does not itself contain the separator, for example percent encoding, markup entities, base64, quoted printable and certain mail encodings. Any escape form that is introduced by the separator character keeps that character present in the raw bytes.

Two. The same byte level mismatch produces false positives as well as misses, which the original framing ignored. Benign prose containing a letter followed by a colon followed by any escaped control character or quote will match a pattern of the shape discussed, because the escape supplies the separator the pattern expects. Operationally this is the failure mode more likely to page someone.

Three. Ordering inside canonicalization was wrong. Removal of format and zero width characters must precede compatibility normalization. Those characters carry combining class zero, so they block canonical composition across themselves, and normalizing first leaves output that is not normalized once they are stripped.

Four, and more consequential for the bounded fixed point. Full case folding is not closed under normalization. At least one Greek precomposed character folds to a three character sequence that recomposes to the original under normalization, so normalize then fold oscillates with period two and never converges. Combined with the reject on non convergence rule, that denies benign text. The remedy is the combined normalization and case folding form defined for exactly this hazard, or a second normalization after folding. Separator unification must remain last, because compatibility normalization synthesizes separators inside some benign single characters, which is a further false positive source, and it can expand a single character into many, which the byte budget must charge after expansion rather than before. Compatibility normalization also does not fold confusable letters from other scripts, so a lookalike drive letter survives it untouched.

Five, a contradiction inside my own proposal. A budget shared globally across the traversal makes the decision depend on the rest of the message rather than on the value alone, so the round trip equality invariant I proposed as a test is false whenever budgets bind, since two encodings of the same value consume different budget. Those invariants must be stated one sided under a disabled budget, with budget accounting tested separately, or they will flake and be switched off.

Six, a wrong justification behind a right conclusion. Absolute per round decode caps compose additively, not multiplicatively. Only a per round expansion ratio compounds. The real multiplier is leaves times rounds, and decompression is the one decode step with a genuinely unbounded ratio.

Seven. Speculative decoding of every string leaf is a quantified false positive generator. Scanning arbitrary decoded binary with a short structural predicate yields on the order of a few spurious matches per megabyte, so any payload carrying an image or attachment denies almost always. Decode only where the declared field grammar says an encoding is expected.

(continued, part 2 of 2)

Eight, two overstatements. Stack exhaustion is catchable in several common runtimes and unrecoverable in others, so the parse time depth limit should be justified by the catch site being nondeterministic and leaving a live copy of the value in arbitrary frames, not by uncatchability. Making deny the default value of the decision type is achievable only in languages with meaningful type defaults. The portable form of that rule is that allow must require an explicit positive token, compared by equality against allow and never by inequality against deny.

Nine, a threat none of the design addressed. The allow or deny decision is itself an oracle. Where any part of the message is influenced by an untrusted party, a visible deny leaks one bit per probe about the host, which undermines the effort spent keeping bytes out of logs. Replacing an offending value with a fixed constant token removes the oracle, and is also the option wrongly excluded earlier, since the argument against redaction refutes partial redaction only and not whole value replacement.