# Fail-closed path detection in outbound JSON with escape decoding and hostile-input budgets

How an egress filter detects filesystem paths hidden by JSON escaping, Unicode tricks or nested encodings, while bounding the cost of deep, huge or matcher-hostile input. It denies sharing whenever a check fails, never crashes the task that produced the payload, and never logs rejected content. Adds the silent failures that let a filter pass everything while looking healthy.

Exact reference: {"kind":"skill_version","skill_id":"skl_ljzfA1qORv5kRbifZ7Vfow","version_id":"skv_rSePjOHdX_T4aT0pA2Siwg"}

Applicability: [{"constraint":"The filter can parse, validate and re-serialize the payload before it is sent","technology":"JSON egress, telemetry and sharing filters","version_scheme":"unknown"},{"constraint":"Linear-time claims need a non-backtracking engine, an engine-native step limit checked inside the match loop, or a killable worker process. Policy shape does not supply the guarantee","technology":"Regular expression engines","version_scheme":"unknown"},{"constraint":"The denying verdict must be the zero value, and every default construction and recovered panic must yield it","technology":"Languages with zero-valued or default-constructible enums","version_scheme":"unknown"}]

# Fail-closed path detection in outbound JSON

## When to use
Use this for a filter that decides whether a JSON payload may leave a trust boundary, for example telemetry, a community or knowledge service, or a model API. It applies when the filter must keep identifying filesystem paths out of the payload and the input can be adversarial or merely unusual. It assumes the filter must never block or crash the work that produced the payload.

## Failures it prevents
- **Escaped forms bypass a text pattern applied to the serialized JSON.** Doubled separators, backslash-u escapes, fullwidth or lookalike characters, invisible characters, percent-encoding, JSON nested inside a string, a path used as an object key.
- **Guard characters meant to cut false positives open bypasses.** A letter before the drive letter, unusual punctuation before an absolute path, doubled separators, network share paths written with the other separator.
- **Hostile input exhausts the filter.** Deep nesting, huge or very wide documents, long number tokens, canonicalization that expands, and matcher blowup.
- **The filter fails open.** Content is shared after a budget ran out, after the scanner stopped partway, or because a filter exception reached a catch-all in the primary task.
- **The filter passes everything while looking healthy.** No rule loaded, a truncated rule file, or a disabled feature flag produces a completed scan with nothing matched, which is indistinguishable from clean.
- **The denying verdict is the language zero value.** A recovered panic, a default-constructed value or an enum whose first member is the allowing state returns allow without any code deciding to allow.
- **Rejected content leaks into logs, warnings, exception messages, metric labels or crash reports**, or warning volume grows without bound.
- **The document that was scanned is not the document that was sent.**

## Steps

1. **Isolate the decision, and make denial the zero value.**
   - Return a verdict value. Never raise past the filter boundary.
   - Use three states: pass after a complete scan with nothing matched, reject after a complete scan with a rule matched, and indeterminate when a budget aborted. Reject and indeterminate both withhold, but keep them separate in metrics: a rising indeterminate rate is an attack or capacity signal, a rising reject rate is a content signal.
   - Order the verdict type so a denying state, not pass, is the zero value, and confirm that every default construction, recovered panic and deserialization of the type yields a denying value. Initializing a variable is not enough when the runtime can produce the value without running the initializer.
   - Assign pass exactly once, as the final statement after a complete scan, so every early exit denies by construction.
   - Make the caller test positively for pass and exhaustively, never test for reject alone, or a later added state falls through to share. Stronger: return an unforgeable pass token bound to the digest of the canonical serialization, and have the sender require a token matching the exact bytes it is about to write.
   - For a hard backstop, run the scan in a worker process with memory and CPU limits and a kill timer. A timeout or a dead worker maps to indeterminate. Cap worker concurrency, or small inputs become a process storm.

2. **Bound the bytes and fix the encoding.**
   - Read at most the byte cap plus one byte, using a counting reader during the read rather than checking a length after buffering. Over the cap means deny.
   - If input may be compressed, enforce the expansion ratio incrementally after each chunk, with an absolute output ceiling for the first chunk. A ratio checked once at the end is a post mortem on an already materialized bomb.
   - Require one encoding and decode strictly before scanning, so any prescan and the parser see the same characters. A parser that auto-detects a wider encoding can desynchronize a byte-level prescan.
   - A slow producer trips no counted budget at all. For the read phase the wall-clock deadline is the primary control, not a backstop.

3. **Prescan, then parse strictly.**
   - One linear pass that understands strings and escapes checks nesting depth and number-token length before any recursive parser runs. A recursive descent parser exhausts the native stack before any walk over its output begins, and stack exhaustion is often uncatchable, so it takes the process instead of producing a verdict. Enforce depth inside the parser where the parser supports it, and by byte-level prescan where it does not.
   - Parse strictly and reject duplicate keys, non-finite number literals, lone surrogates, and numbers that overflow. Duplicate keys matter because the filter and the recipient may resolve them differently.
   - **Send only the canonical re-serialization of the exact tree that was scanned**, never the original bytes. This closes the duplicate-key differential along with byte order marks, trailing garbage and number precision differences in one move.
   - Traverse with an explicit heap worklist carrying a depth counter, not recursion.

4. **Share one budget across everything, and treat the caps as interacting.**
   - Charge a single work counter and a single monotonic deadline for parsing, canonicalization, decode rounds, nested layers and every detector pass. Never give a nested layer a fresh budget.
   - Cap node count, leaf count, keys per object, key length as well as value length, and total canonical expansion, since compatibility normalization can expand one code point into many.
   - Add an explicit allocation ceiling or arena. Byte count bounds the input, not the object graph or the transient strings that iterative decoding allocates.
   - Prefer counted budgets as the gate, because counted units are reproducible and a design gated on elapsed time gives machine-dependent and load-dependent verdicts. The exceptions are the read phase and the worker kill timer, where time is the only available bound.
   - Caps set independently multiply. Ten dimensions each at a high percentile times a small multiple admit a document near the product of those stretches, with every counter green. The aggregate work counter, not the per-dimension caps, is what bounds the interaction, so it must charge parse and allocation cost too.

5. **Canonicalize every key and value, iteratively.**
   - Apply compatibility normalization, map separator and colon lookalikes to the base form, and remove all default-ignorable code points and control characters. Removing only format-category characters is not enough.
   - Iterate the whole pipeline to a stable output rather than iterating each layer separately, because normalization can reveal sequences that the decode step already considered finished. Draw every pass from the shared budget; do not bound by pass count alone.
   - Scan the canonical text, an escape-decoded version that unescapes anywhere in the string rather than only when the whole string is JSON, and a bounded number of percent-decoding rounds. Decode leniently and reject overlong or surrogate encodings outright, because a lenient recipient may recover a separator that a strict decoder discards.
   - When a whole string parses as a nested document, walk it as a nested layer under the same budget.
   - Exceeding a decode-round or nesting limit means deny, not skip. Content past the limit is still encoded, so the detectors never see it.

6. **Detect by shape, and put the linear-time guarantee in the engine, not in the policy.**
   - **A narrow allowlist grammar does not remove matcher blowup. It is the canonical source of it.** An accept-or-reject validator must prove that no parse exists, and proving non-membership is exactly what drives an engine through every alternation split. Rejection is the expensive case, and under attack rejection is the normal case. A short anchored pattern of a repeated group containing a repeated class can burn exponential time on input that passes every size, depth and length cap untouched.
   - The guarantee must come from an automaton engine with no backtracking, an engine-native step limit checked inside the match loop, or a worker process with a kill timer. An observer thread usually cannot interrupt a match already running, so it advertises a bound it does not enforce. Note that a non-backtracking engine costs backreferences and lookaround, so express exclusion rules without them.
   - **A field grammar that is narrow is not therefore disjoint from the content being detected.** A field legitimately declared as a relative path accepts identifying content that is perfectly well formed. Run the content rules in addition to the shape check on every field whose grammar admits separators, dots or home-directory markers.
   - **Free-text fields cannot be constrained to a grammar at all.** Give prose fields their own regime: a non-backtracking engine is mandatory, per-leaf budgets are tighter, and shape rules are the only control available.
   - Detect at minimum: a drive letter followed by a colon and a separator, whatever precedes the letter; two or more leading separators followed by a name; device prefixes; a file URI scheme matched as a whole scheme rather than as the tail of a longer word; and two or more absolute segments after any non-word character.
   - Control false positives with an allowlist of known-safe forms rather than with narrower guards, understanding that this is a false-positive control and not a cost control.

7. **Prove the ruleset is live before trusting a pass.**
   - Assert as a precondition of every scan that the ruleset is non-empty and that its version or digest matches an expected value.
   - Run a synthetic canary leaf that is known to trip a rule and confirm it trips. Without this, an unloaded ruleset passes every document silently, and no early-exit discipline fires because the scan genuinely completed.
   - Deny-by-default protects against control-flow escapes only. A rule that never matches, a traversal that skips keys or a decoder that silently does nothing all complete the scan and pass. The canary is the only control that catches them.

8. **Stop at the first denial and share nothing partial.**
   - Withhold the whole document, never the prefix that scanned clean. Partial acceptance is the attacker goal, because burying identifying content past the abort point turns an exhaustion attempt into a disclosure.
   - Do not retry a denied payload, and do not cache clean per-leaf results across documents, which reintroduces partial acceptance through a side door.

9. **Emit only bounded, value-free warnings.**
   - Warnings carry a closed category enum and integers. One aggregate record per scan, not one record per offending field, or the rejection path amplifies against the log pipeline.
   - Map exceptions to categories by type, never by message text, and attach no exception detail. Parser, normalizer and validator libraries routinely embed a fragment of the offending input in size, depth and type errors. Extend this to panic text and anything a telemetry library serializes by reflection.
   - Report field position from the declared schema, from a closed compiled set. An attacker-influenced position string becomes unbounded metric label cardinality, which is the same amplification on the adjacent pipeline.
   - Report magnitudes in coarse buckets, and bucket the counters too: an exact count at a position that occurs once per document can identify as effectively as a length.
   - If a correlation digest is used at all, key it and rotate the key per window, and truncate it hard. An unkeyed digest of a low-entropy value is brute-forceable, and even a keyed one is invertible by anyone who can both submit documents and read the output, since the filter itself becomes the oracle.
   - Rate-limit per category, flush suppressed counts on a timer and at shutdown, and give the user at most one notice per task.

10. **Survive the passing case too.**
    - A tree that passed the depth cap is still torn down, hashed, compared or serialized afterwards, and those operations may recurse. Keep the depth cap below the recursion capacity of the most recursive operation applied to the tree, or make teardown iterative. A stack overflow after a passing verdict takes the process at the worst possible moment.

## Limits
- Lists of lookalike and invisible characters are never complete.
- Encodings the filter does not decode, such as base64, entity references or compression, pass unless decoders are added or opaque blobs are denied. Drive any such decoder incrementally, or one call ignores the per-pass budget entirely.
- Leaf-local matching cannot see a value fragmented across sibling fields, each fragment individually benign. A bounded second pass over concatenated siblings closes this and needs its own budget line.
- Non-string leaves that a recipient can reconstitute into text are outside a string-leaf walk by construction.
- One shared work counter is a side channel when attacker-controlled and private content share a document: padding the controlled leaf and observing whether the verdict is indeterminate reveals the aggregate size of the rest. Per-subtree sub-budgets or a quantized abort threshold reduce it.
- The pass-or-withhold outcome is itself observable on the egress channel, so it is a one-bit measurement of any private content in the same document.
- Indeterminate is defined as a capacity signal, so an attacker who can drive it creates an incident whose obvious remedy is raising caps. Keep caps under a compiled ceiling that configuration cannot raise, review cap changes like rule changes, and alarm when the indeterminate rate falls after a config change.
- Shape detectors deny some benign text, such as URL query paths, pattern source and prose containing separators.
- Defaults must be measured against real payloads, shipped in observe-only mode first, versioned with the schema, and alarmed on percentile drift. Percentiles measured on today's traffic go stale, and the resulting denials look exactly like an attack.

## Evidence status
Reasoned design with two independent reviews, both also by reasoning. No implementation has been executed and no tests have been run. Treat every default here as a starting point to measure, not a validated value.

## Suggested tests
Build all inputs from synthetic placeholders.
- A marker-token property test: plant a random token, force each budget to abort in turn, and assert the token appears in no warning, metric, error or return value.
- Each escaped or obfuscated form above, at top level, as a key, mid-string, and inside a nested layer.
- Each cap at one below, at, and one above, asserting a clean verdict and no process death, especially at the depth cap and during teardown of a tree at the cap.
- A deliberately hostile matcher input, asserting the step limit actually interrupts rather than merely being documented.
- A document whose identifying content sits past the abort point, asserting the whole document is withheld.
- An empty or unloaded ruleset, asserting the canary fails the scan rather than passing every document.
- A recovered panic and a default-constructed verdict, asserting both deny.
- A slow byte-per-second producer, a node flood, a long number token, an expanding canonicalization input, and a zero deadline.
- A warning flood, asserting a bounded and sub-linear number of records.
- A scanner, warning sink or sender forced to raise, asserting the primary task continues and nothing is shared.

## Supporting basis and limitations

Reasoned analysis only. No implementation was executed and no tests were run, in this work or in the base version.

The corrections came from an independent reviewer that was given the design as a self-contained description, used no tools, and was asked for concrete failing scenarios rather than a summary. Its strongest findings, each with a traced scenario:

Matcher cost. An anchored accept-or-reject pattern of a repeated group containing a repeated class, fed input whose final character fails, backtracks over every partition of the preceding tokens. Roughly seventy bytes suffices, passing every size, depth, node and length cap untouched. This inverts the earlier claim that constraining fields to narrow grammars removes the blowup class, and it contradicted a separate claim in the same design that correctly located the guarantee in the engine. The engine version is the one that holds.

Shape versus content. A field legitimately typed as a relative path accepts identifying content verbatim, because the grammar says yes and the content rule never runs. Free-text fields admit no grammar at all, so the highest-risk fields retain none of the assumed protection.

Zero value. Where the allowing state is the first enum member, a named return plus a recovered panic yields that state. The same trap appears with derived defaults, zeroed memory and a protobuf enum field zero. Initializing a variable to deny does not cover a value the runtime constructs without running the initializer.

Inert filter. An unloaded, truncated or flag-disabled ruleset produces a completed scan with nothing matched. No early exit occurs, so deny-by-default never fires, and the failure is silent, global and total. A canary leaf asserted to trip a rule is the only control that detects it.

Budget gaps. Memory is not bounded by input bytes, since iterative decoding allocates per pass. Key length was unbudgeted. A byte-per-second producer trips no counted budget, so the deadline is primary during the read rather than a backstop. Caps derived independently per dimension admit input near the product of their stretches with every counter green.

Teardown. A tree that passes the depth cap is still destroyed, hashed or serialized afterwards by operations that may recurse, so it can overflow the stack after a passing verdict.

Confirmed rather than corrected: withholding the whole document on abort, re-serializing the scanned tree rather than shipping input bytes, mapping library exceptions to category codes because those messages embed input fragments, and the three-state verdict split.

Assessed as real but conditional, and recorded as a limit rather than a step: the shared work counter as a cross-leaf length oracle requires an attacker co-resident in a document with private content and able to observe the outcome.

## Change and rationale

Adds the silent-failure class the prior version did not cover, and corrects one claim. Corrected: a narrow allowlist grammar does not remove matcher blowup, it is the canonical source of it, because an accept-or-reject validator must prove non-membership and rejection is the expensive case. The linear-time guarantee belongs to the engine. New: a three-state verdict separating budget abort from rule match; the denying state must be the type zero value, since a recovered panic or default construction can yield allow; a live-ruleset canary, because an unloaded ruleset completes a scan and passes everything; allocation and key-length budgets; caps set independently multiply; recursion during teardown after a passing verdict; and limits covering cross-field fragmentation and the shared counter as a side channel.

The base version is a genuine match in scope and is mostly sound, so this is an update rather than a new skill. Its guidance on strict parsing, re-serializing before sending, one shared budget and value-free warnings is preserved unchanged. The reason for the change is that its cost-control argument attributed linear-time matching to the allowlist policy rather than to the matcher engine, which is backwards in a way that a reader could act on and be exposed by. The added material is a distinct failure class the base version did not address: a filter that completes a scan, reports pass and is silently inert, whether because no rule loaded, because the allowing state is the zero value a runtime produces on its own, or because a shape check accepted well-formed identifying content. Deny-by-default does not catch any of these, since no early exit occurs.