Validate Streaming UTF-8 Limits Without Treating Chunks as Text Boundaries

Use this procedure when a network body or field arrives incrementally and must satisfy both an encoded byte ceiling and a semantic Unicode limit. The semantic limit might count code points or grapheme clusters. The byte limit applies to a specifically named representation, such as compressed transport bytes or decompressed UTF-8 body bytes.

The central rule is that network chunks are delivery artifacts. They are not character, normalization, or grapheme boundaries.

Freeze the two boundaries

Before processing data, record:

• The exact stage whose bytes are limited: before decompression, after decompression, after framing removal, or another named boundary.
• Whether the byte maximum is inclusive.
• The semantic counting unit: Unicode code points, grapheme clusters, or a contractually named code-unit scheme.
• Whether normalization is required and whether it occurs before semantic counting.
• The invalid UTF-8 policy. For validation, prefer strict rejection rather than replacement characters.
• Whether the semantic and byte errors are reported independently or by documented precedence.

If both compressed input and decompressed output have ceilings, maintain two byte budgets at those two stages. One counter cannot represent both contracts.

Process each chunk without decoding it in isolation

For every chunk at the byte-limited boundary:

1. Determine the remaining byte budget before adding the chunk.
2. Reject for encoded size if the chunk length exceeds the remaining budget. Comparing against the remainder avoids overflow in the accumulated total.
3. Add the accepted chunk length to the byte total.
4. Feed the same bytes to one strict incremental UTF-8 decoder.
5. Count only complete Unicode output emitted by that decoder.
6. Preserve all decoder, normalization, and segmentation state for the next chunk.

Do not decode each chunk as a standalone string. A valid multibyte scalar may begin in one chunk and end in another. Do not use a replacement decoder: replacement can turn malformed bytes into apparently valid text and can change the semantic count.

Do not re-encode decoded text to reconstruct the received-byte count. Re-encoding loses evidence about malformed input and can produce a different representation after replacement, normalization, or other transformations.

Count semantic units with matching streaming state

For a code-point limit, count complete scalar values emitted by the incremental decoder. A partial multibyte sequence contributes bytes to the byte budget but contributes no code point until completed.

For a grapheme-cluster limit, do not sum the grapheme count of each decoded chunk. A base character, combining mark, joiner sequence, or regional-indicator sequence can cross a chunk boundary. Use a streaming segmenter that retains the required suffix state, or buffer the decoded value until it can be segmented correctly.

Apply the same rule to required Unicode normalization. Use a correct streaming normalizer with retained state, or buffer until finalization. Normalizing each chunk independently is not equivalent to normalizing the complete value.

Reject as soon as either accumulated limit is exceeded when the contract permits early rejection. Keep the counters and error identities separate even if the implementation stops at the first failure.

Finalize explicitly at end of stream

End of stream is a validation event, not merely the absence of another chunk.

Finalize the incremental decoder in strict mode. Reject if it retains an incomplete UTF-8 prefix or reports malformed input. Then flush any required normalizer and segmenter, count their final output, and apply the semantic ceiling one last time.

An incomplete final sequence can remain within the byte budget and still be invalid UTF-8. Do not misreport that case as a character-limit or byte-limit violation.

Prove chunk-boundary invariance

Use one fixed byte sequence and deliver it through several partitions:

• One complete chunk.
• One byte per chunk.
• A split at every possible byte boundary.
• Several deterministic irregular partitions.
• Empty chunks if the transport permits them.

For every partition, assert the same byte total, decoded value, semantic count, accept-or-reject decision, and error identity. Partitioning may change timing, but it must not change meaning.

Include cases exactly at each inclusive ceiling and just beyond it. Also include malformed input and a valid multibyte sequence truncated at end of stream.

Reasoned examples

These examples are reasoned from ordinary UTF-8 and are not executed tests.

• Precomposed é is encoded as hexadecimal C3 A9. Delivered as C3 and then A9, it consumes two bytes and produces one code point only after the second chunk.
• Grinning face is encoded as hexadecimal F0 9F 98 80. Delivered one byte at a time, it consumes four bytes and produces one code point after the fourth byte.
• Bytes E2 82 at end of stream consume two bytes but form an incomplete three-byte sequence. A strict finalization rejects malformed UTF-8 even when both configured size ceilings would otherwise pass.
• ASCII e followed by combining acute accent uses three UTF-8 bytes and two code points but commonly forms one grapheme cluster. If the two scalars arrive in different chunks, summing per-chunk grapheme counts can produce the wrong semantic result.

Common failures this procedure detects

• Treating every transport chunk as a complete string.
• Counting decoded characters as a substitute for received bytes.
• Re-encoding decoded text to estimate the original byte total.
• Resetting the decoder, normalizer, or grapheme segmenter between chunks.
• Accepting an incomplete multibyte prefix at end of stream.
• Letting integer addition wrap before comparing with the byte ceiling.
• Counting compressed bytes for a decompressed-byte contract, or the reverse.
• Producing different validation results for different partitions of identical bytes.

Evidence status

The numeric examples and expected outcomes above are reasoned results, not runtime observations. No tests or captures were executed for this skill. An implementation should record its decoder mode, normalization policy, segmentation library, counted byte stage, partitions exercised, and observed results when it performs actual verification.