Validate UTF-8 Byte Limits Independently from Unicode Character Counts

Use this guidance when user-visible text is placed in a request body, protocol field, queue message, event, or other network payload with a size ceiling.

Core rule

Validate semantic length and encoded size as separate invariants.

A semantic text rule must name what it counts: grapheme clusters, Unicode code points, or language-specific code units. A network-size rule must name which encoded bytes it counts. Never use a string length result as a substitute for UTF-8 byte length.

Define the boundary before implementing

Record all of these details from the contract:

• Whether the limit applies to one field, the serialized body, or the full protocol message.
• Whether the unit is characters, code points, code units, grapheme clusters, bytes, or octets.
• Whether the maximum is inclusive.
• Whether Unicode normalization is required and, if so, which form and when it occurs.
• Whether escaping, envelope fields, headers, framing, or compression are inside the measured boundary.

Do not count headers, framing, or compressed bytes unless the contract includes them. Do not assume a service limit uses the same boundary as an HTTP Content-Length value.

Validate in the production order

1. Apply only the canonicalization or Unicode normalization required by the contract.
2. Enforce any semantic text limit using the explicitly chosen Unicode counting unit.
3. Serialize the exact request representation with the production serializer and its real escaping options.
4. Encode the scoped representation as UTF-8.
5. Count the resulting bytes with a byte-length API.
6. Accept an inclusive limit only when the measured byte count is at most the maximum.
7. If shortening is allowed, remove complete semantic units, serialize again, and recount. Never cut an arbitrary UTF-8 byte slice because that can end inside an encoded scalar value.

For a field-level byte limit, encode and count the final field value at the point specified by the contract. For a whole-body limit, count the final serialized body rather than summing source-string lengths.

Boundary examples

The following examples deliberately separate visible text from encoded size:

• ASCII letter A is one Unicode code point and one UTF-8 byte.
• Precomposed é is one code point and two UTF-8 bytes.
• A common CJK ideograph is one code point and three UTF-8 bytes.
• Many emoji are one code point and four UTF-8 bytes.
• Decomposed e followed by a combining acute accent is two code points, one commonly perceived grapheme cluster, and three UTF-8 bytes. It is not byte-equivalent to precomposed é.
• A JSON serializer may emit a character directly or emit an ASCII escape. Measure the serializer output because quote escaping, backslash escaping, control-character escaping, and an ASCII-only mode can change the body size.

Concrete inclusive field boundary

Assume a contract applies an inclusive four-byte ceiling to the final UTF-8 encoding of one field, with no required normalization or escaping after validation.

• A single emoji that encodes as four UTF-8 bytes contains one Unicode code point and is accepted because its byte count equals the ceiling.
• The same emoji followed by ASCII A contains two Unicode code points and occupies five UTF-8 bytes, so it is rejected even if a separate character limit would allow two code points.

This example establishes equality and one-byte-over behavior only for that field-value boundary. It does not establish the size of a containing JSON body, a serializer escape form, a compressed representation, or a framed protocol message. Measure those representations separately when the contract limits them.

Normalization must be a contract decision, not a hidden tactic for making a value fit. Canonically equivalent strings can have different code-point and UTF-8 byte counts.

Focused verification

Test the exact limit and one byte on either side when those cases are constructible. Include:

• An ASCII-only baseline.
• Two-byte, three-byte, and four-byte UTF-8 scalar values.
• Precomposed and decomposed forms of the same visible text.
• Combining marks and multi-code-point grapheme clusters.
• Quotes, backslashes, control characters, and any serializer-specific escaping mode.
• Empty payloads and fixed envelope overhead.
• A field that fits by semantic count but fails by encoded byte count.
• A whole body whose fields fit individually but whose serialized envelope exceeds the body limit.
• Compression or framing variants only when the documented limit includes that stage.

Prefer an integration assertion against the exact byte buffer handed to the relevant transport boundary. A useful invariant is that every accepted representation measures within the declared byte ceiling at that boundary.

Failure reporting

Report the two measurements separately when both rules exist, for example semantic units used and UTF-8 bytes used. Name the counting unit and byte boundary in diagnostics so callers can correct the right constraint.

Evidence status

This guidance follows from UTF-8 encoding and serialization semantics. The numeric examples, including the inclusive four-byte field case, are reasoned encoding results. No executable tests, packet captures, or production measurements were run for this skill proposal; each implementation must verify its concrete runtime, serializer, and protocol contract.