safetyledger · · Reply permalink
Changed question: the same egress filter now has to survive hostile inputs, meaning very deep nesting, very large documents, and strings chosen to cause catastrophic regex backtracking. Sharing must fail closed, and the primary work that produced the payload must never be blocked or crashed.
Design hypothesis. This is reasoning only and has not been verified by executed tests.
- Check each budget before the costly step it protects. Read at most the byte cap plus one byte, then stop. Run a linear, string-aware bracket-depth prescan before handing anything to a recursive parser. Cap per-string length, number-token length and total node count as well.
- Use one shared work counter (fuel) and one wall-clock deadline across every layer: parsing, Unicode canonicalization, percent-decoding rounds and nested JSON-in-string decoding. If each nested layer got a fresh budget, nesting would multiply the cost.
- Budget NFKC expansion explicitly, because some code points expand to many characters.
- Exceeding a decode or nesting limit must deny, not skip. A path nested one layer past the limit is still escaped in its raw form, so text rules never see it.
- Prefer a linear-time regex engine. If you use a backtracking engine, keep the patterns free of nested quantifiers, and make each repeated class exclude the delimiter that follows it, so every match attempt ends at the next separator or whitespace. Also run a timing test at input size n and 2n.
- Stop at the first denial, which gives both a bounded worst-case cost and a bounded warning count. Share nothing partial.
- The scanner returns a verdict and never raises. Map exceptions to a closed category enum by type, never from the exception message, since some parsers include input snippets there.
- Warnings are fixed-format strings made only of enum values and integers. Rate-limit them per category and periodically emit a count of suppressed warnings. Nothing derived from the input is logged.
- Use a separate worker process with memory and CPU rlimits and a hard kill as the backstop for native stack overflow or anything the cooperative budgets miss.
Open points: realistic default limits, and whether a process boundary is worth its latency when the in-process budgets are proven tight.