# Privacy filter for structured payloads: decode-validate-freeze to prevent encoding evasion and diagnostic leaks
## When to use
Use this when outgoing structured data must be scanned for sensitive patterns before transmission to external systems, logging infrastructure, or untrusted consumers. Applies when the serialized form can differ from semantic content through escaping, encoding, or nesting, and when validation failures must be observable without leaking the rejected values.
Common triggers: JSON payloads containing potential filesystem paths, usernames, internal URLs, or other identifying data being sent to third-party APIs, client-side logging, or analytics systems.
## The dual failure mode
Privacy filters commonly fail in two ways:
1. **Encoding evasion**: A regex written for plain text misses encoded forms. A filesystem path can appear with backslashes doubled by JSON string encoding, percent-encoded as URL data, Base64-encoded, or nested across multiple JSON-in-JSON layers. The pattern matches the wire bytes while the consumer decodes them, so the two disagree.
2. **Diagnostic leakage**: When validation rejects a payload, logging the rejected value for debugging exposes the sensitive data the filter was meant to protect. Even metadata like field paths, value hashes, or lengths can leak information or enable verification attacks.
## Procedure
### 1. Define the pattern scope explicitly
Document exactly what counts as a sensitive pattern. For filesystem paths:
- Unix absolute paths starting with forward slash
- Windows paths with drive letters or UNC format
- **Relative paths including those without leading separators**: forms like DIRNAME SLASH FILENAME, DOT DOT SLASH DIRNAME, or single segments that could reveal repository structure
- Home directory expansion forms
Explicitly list what does NOT match to avoid false positives: semantic versioning strings, URL paths in legitimate external links, code examples in documentation fields.
**Platform consideration**: If validating cross-platform payloads, include both Unix and Windows patterns even when the validator runs on one platform. If platform-specific, document which path forms are checked.
### 2. Multi-layer decoding in bounded rounds
Parse the structured payload. **Validate both object keys and values** — object keys are strings and can carry identifying data that value-only validation misses.
For each string value and each object key:
1. Decode JSON escapes to canonical form
2. Attempt URL decoding if the string matches percent-encoding patterns
3. Attempt Base64 decoding if the string matches Base64 alphabet and padding
4. **Check for nested JSON strings**: if a decoded string parses as valid JSON, treat it as a nested layer
5. Limit total decode rounds to 3 maximum to prevent infinite decode loops
Track each decode layer. If decoding reaches the round limit, treat the value as potentially evasive and apply the strictest validation.
**Nested format policy**: Either document that nested JSON-in-JSON strings are not recursively validated beyond the round limit, or recursively parse and validate with strict depth accounting against the total nesting budget.
### 3. Resource budgets enforced before and during validation
Apply three hard limits in sequence:
- **Byte size cap first**: Reject payloads exceeding a configured maximum at ingress before parsing. Check raw byte size, then if the payload is compressed, apply the limit again after decompression before parse. One megabyte uncompressed is typical.
- **Parsing depth limit during parse**: Cap nesting depth at 32 levels. Count during parsing, not via language stack depth, so overflow surfaces as a controlled denial rather than an exception.
- **Work ledger during validation**: Charge per decoded character, per decode round, and per pattern check. Use linear-time pattern matching only: no backreferences, no lookaround, no nested quantifiers. Backtracking regex engines hide cost inside one call and cannot be metered reliably.
On any limit violation: reject immediately, emit only the limit type violated, do not process further.
### 4. Normalize then validate on canonical decoded values
Before pattern matching, normalize paths to canonical form:
- Collapse repeated separators
- Remove current-directory segments
- Handle mixed separators if validating cross-platform payloads
Apply patterns to the normalized decoded string, not the serialized wire bytes. Use a regex engine with guaranteed linear time complexity or explicit backtrack limits, not wall-clock timeouts which are platform-dependent and unreliable.
Prefer: RE2, Rust regex crate, or DFA-based engines. Avoid: default Python re, JavaScript RegExp, PCRE without backtrack caps.
### 5. Unknown field handling
Define an explicit policy for fields not in the schema:
- **Fail-closed strict**: Reject payloads containing unknown fields
- **Validate unknown fields generically**: Apply the same pattern checks to unknown string fields as to known ones
- **Fail-open**: Allow unknown fields through without validation (document this as a potential bypass)
### 6. Immutable validation to prevent TOCTOU
If validation runs asynchronously in a worker process or thread:
1. Serialize the payload to an immutable byte snapshot before validation starts
2. Validate the snapshot
3. Transmit exactly the validated bytes, never re-serialize the original object
This prevents malicious code from mutating the payload between validation and transmission.
### 7. Aggregated-only logging
On rejection, log ONLY:
- Pattern category matched from a closed enum
- Count of violations for this category in this payload
- Timestamp and correlation ID
Do NOT log:
- Field names or paths: attacker-controlled and reveal structure
- Value hashes: enable candidate verification attacks
- Value lengths: side channel that narrows search space
- Character class summaries combined with length: together they leak information
- Any substring or excerpt of the rejected content
For debugging false positives, rely on the correlation ID to match with sender-side logs where the full context is already known.
### 8. Bounded warning cardinality
Cap unique warning detail per time window. For example:
- Maximum 50 distinct category emissions per hour
- After cap: increment a suppression counter but emit no new log lines
- Categories are a closed enum, not derived from payload content
This prevents diagnostic explosion from becoming its own denial-of-service vector or leak channel.
### 9. Fail-closed with primary task continuation
Run the validation filter in a worker with a wall-clock deadline and memory cap. The outbound sender requires an explicit allow-transmit handle that only a successful validation produces.
On worker crash, timeout, or limit violation:
- No transmit handle is produced
- Nothing can be sent
- The supervisor returns control to the caller immediately
- Primary request processing continues with a safe denial outcome
Mark denied payloads to prevent retry loops from burning CPU. Implement a circuit breaker: after N consecutive crashes or timeouts, disable the validation worker for a cooldown period and fail all requests closed during that window.
## What this prevents
**Encoding evasion**: Multi-layer canonical decoding catches encoded forms that evade surface-level patterns.
**Normalization bypass**: Path normalization before matching prevents repeated separators and current-directory segments from evading detection.
**Object key bypass**: Validating both keys and values prevents identifying data from hiding in property names.
**Nested format bypass**: Bounded recursive decoding prevents nested JSON-in-JSON or embedded encodings from carrying unvalidated content.
**Resource exhaustion**: Hard budgets on size, depth, and work prevent malicious payloads from causing denial of service through the validator itself. Size limits after decompression prevent compression expansion attacks.
**Diagnostic leakage**: Aggregated-only logging ensures rejection events are observable without exposing the sensitive content that triggered the filter, and without creating side channels through metadata or combined signals.
**Race conditions**: Immutable snapshot validation prevents mutation between check and transmission.
**Operational collapse**: Fail-closed worker isolation with circuit breaking ensures validation complexity or instability cannot block the primary request path.
## Limitations and known gaps
This procedure is based on reasoned analysis of common encoding evasion and diagnostic leak patterns, not on executed tests or production deployment data.
**What it does not cover**:
- Sensitive data in non-string types: integers, booleans, or binary data
- Information leaked through field presence or absence rather than values
- Timing side channels where validation duration varies with content
- Allowlist mechanisms for legitimate mentions of paths in documentation
- Pattern maintenance as new encoding schemes emerge
- Cross-boundary consistency when different systems use different parsers with different normalization rules
**Deployment prerequisites**:
- A regex engine with linear-time guarantees or explicit backtrack limits
- Immutable data structures or copy-on-validate discipline in the validation worker
- Monitoring for circuit breaker activation and suppression counter growth
- Clear documentation of what patterns match so callers know what will be blocked
- Decision on unknown field policy appropriate to the trust model
**When not to use**:
- For allowlist validation where the permitted set is small and enumerable: exact matching is simpler
- When the consumer and validator use different parsers with different lenience: results will disagree
- For real-time high-throughput paths where validation latency is unacceptable: consider sampling or async post-transmission auditing instead
## Relation to existing guidance
This builds on established patterns for resource budgeting, fail-closed validation, and encoding normalization, but combines them specifically for privacy-sensitive structured data where diagnostic transparency usually conflicts with data protection. The multi-layer decode procedure, path normalization discipline, object key validation, and aggregated-only logging are the elements addressing gaps found in simpler approaches.