Design question about an outbound privacy filter that scans JSON payloads for identifying filesystem paths. A regex written for ordinary text misses Windows-style paths once JSON escaping doubles each backslash, and further evasion is possible through unicode escapes, forward-slash variants, or splitting a path across nested fields. Known evidence: matching against the raw serialized document is fragile because the same logical string has many wire encodings. Open questions: the right layer to validate at (decoded values after parsing rather than serialized bytes), how to normalize candidates before matching, and how to report a rejection with enough diagnostic value for operators without ever persisting the rejected string itself. Looking for prior patterns on canonicalization before matching, structural walks of parsed JSON, and safe rejection telemetry such as field paths, rule identifiers, lengths, and salted hashes.
Detecting escaped Windows paths in outgoing JSON without logging rejected values
Resolution, by reasoning only, no tests executed. The defect is a layer mismatch rather than a weak pattern: the filter inspects serialized bytes while the rule was authored for decoded text. In the wire form every literal backslash is doubled, so a rule shaped as separator followed by a run of non separator characters consumes the first backslash and then fails on the second. One nuance worth keeping: a unicode escape of the backslash still begins with a backslash, so a byte level scan for the separator does fire; the real evasions of a byte scan are encodings that contain no separator at all, such as percent encoding, base64 or markup entities, plus forward slash Windows forms, network share prefixes and values split across fields. The fix is to parse strictly, walk the tree, canonicalize each string leaf in a convergent order, constrain schema fields with positive grammars, run structural path heuristics on free text, and re-emit from the validated tree rather than forwarding the original bytes. Rejection records carry schema position, rule id, coarse length bucket, boolean class flags and a keyed digest under a rotating key, never the value, and parser error text must also be suppressed since it echoes input. Verification is a property test: plant a marker and assert it never appears in any log, metric label or error text. The existing skill on validating egress payload strings after decoding already covers every point above in more depth, so no new skill is warranted.
Refinement, reasoning only, no tests executed. The question extended to hostile inputs: very deep nesting, very large bodies and matcher inputs that trigger pathological backtracking. Resolution: gate outside in with a byte cap on a bounded reader and a linear nesting pre-scan that tracks string state before any parse call, since a standard parser consumes the whole document before the walk begins. Enforce depth during parse rather than by catching stack exhaustion. Keep per dimension caps only as cheap early rejects, because independent ceilings on depth, nodes, bytes and string length multiply, and meter everything from one global fuel counter charged per node, per normalized code point and per matcher step, in counted units rather than elapsed time. Use a linear time matcher where possible; a step limit only converts blowup into a spurious deny. Initialize the verdict to deny, assign allow once at the end, map exceptions to deny excluding cancellation, and return a three way verdict so budget aborts are distinguishable from detections in metrics. Buffer fully and validate before releasing anything, because a stream abort can arrive after the prefix has flushed. The primary task treats any non allow verdict as do not share and continues; bound retries so a poisoned field cannot become a hot loop. Diagnostics are one aggregate record per pass with a closed category enumeration, which limit fired first, capped distinct categories with a truncation flag, deduplication keyed on category and schema position, wildcarded unrecognized keys, and a scrubbing boundary on the exception handler since parse and conversion errors retain input as attributes. One noted tension between the two existing skills: one suggests a keyed digest under a rotating key for correlation, the other warns a long lived key makes that a stable pseudonymous identifier. Reconcile by keeping the key lifetime to the correlation window or omitting the digest. The second existing skill already covers this material in depth, so no new skill is warranted.