Fit Unicode Text to a UTF-8 Payload Budget Without Splitting Graphemes

Use this procedure when a network contract permits shortening a text value that exceeds a byte ceiling and the product must preserve the longest meaningful prefix. Continue to validate any Unicode character limit separately; fitting the byte budget does not prove that the semantic limit passes.

Freeze the shortening contract

Record these decisions before truncating:

• The exact representation whose bytes are limited: the final field value, serialized body, framed message, or another named boundary.
• All required normalization, escaping, envelope, and encoding steps before that boundary.
• The semantic unit that may be removed. For user-visible text, use extended grapheme clusters unless the contract names a different unit.
• Whether an omission marker is required and its exact text.
• Whether the byte maximum is inclusive.
• Whether an empty retained prefix, a marker by itself, or complete rejection is allowed.

An omission marker consumes the same measured budget as the retained text. Do not append it after validation.

Build one exact measurement function

Define one candidate builder that accepts a retained semantic prefix and performs the production transformations in their contractually required order. It must insert the real omission marker when shortening occurred, place the result in the real envelope, serialize with production options, encode the scoped representation, and return its byte count.

Keep semantic counting separate. The byte measurement function answers only whether that exact candidate fits the named encoded boundary.

Select a safe prefix

1. Apply only normalization or canonicalization required before segmentation.
2. Segment the value into the contractually allowed removal units. Never use code-unit indexes as if they were grapheme boundaries.
3. Measure the unshortened production candidate. Return it unchanged when it fits both the byte rule and the separate semantic rule.
4. Measure the shortest allowed shortened candidate, including the omission marker. When the contract permits an empty retained prefix, the marker-only candidate is accepted if its exact measured size equals an inclusive ceiling. If that candidate exceeds the byte ceiling, fail instead of silently dropping the marker or changing the contract.
5. Evaluate candidates only at complete semantic-unit boundaries.
6. Starting with the longest permitted prefix, build and measure the exact production candidate. Return the first candidate that fits.
7. If no candidate fits, follow the documented empty-value or rejection policy.
8. Run the independent semantic-limit validator on the selected result and report the two measurements separately.

Searching from longest to shortest is correct even when final encoded size is not monotonic, although it can require many serializations. A binary search or a stop-at-first-failure forward scan is valid only after proving that adding a semantic unit cannot reduce the measured size for the exact transformation pipeline. Do not assume monotonicity when compression, content-dependent envelopes, conditional fields, deduplication, or another context-sensitive transform is inside the measured boundary.

Do not truncate encoded bytes

Never slice a UTF-8 buffer at the remaining byte count and decode the prefix. The slice can end inside a multibyte scalar. Even a scalar-safe cut can split a grapheme cluster, such as a base letter from a combining mark or part of a joiner sequence.

Truncate the semantic value at an allowed boundary, rebuild the production representation, and measure again.

Reasoned examples

These are reasoned encoding examples, not executed tests.

Marker-only equality boundary

Assume the contract permits an empty retained prefix, requires a single ellipsis character as the omission marker, applies an inclusive ceiling to the final field value, uses ordinary UTF-8, and performs no later normalization or escaping.

• The ellipsis marker occupies three UTF-8 bytes.
• Under an inclusive three-byte ceiling, the marker-only shortened value fits exactly and is accepted.
• Under an inclusive two-byte ceiling, the same required marker does not fit. No legal shortened candidate exists, so the procedure rejects instead of removing the marker.

This example establishes only the marker-only equality and over-limit decisions at that final field-value boundary. It does not decide whether a product should allow an empty retained prefix, and it does not establish the size of a containing serialized body or framed message.

Longest-prefix example

Assume an inclusive five-byte ceiling on the final field value, ordinary UTF-8, no normalization or escaping, and grapheme-cluster truncation.

• The value made of ASCII A, grinning face, and ASCII B occupies six bytes: one plus four plus one.
• Without an omission marker, the longest fitting grapheme prefix is ASCII A followed by grinning face, which occupies exactly five bytes.
• With a single ellipsis character as the required marker, the marker occupies three bytes. ASCII A followed by the marker occupies four bytes, while ASCII A followed by grinning face and the marker occupies eight bytes. The longest fitting shortened result is therefore ASCII A followed by the marker.

A decomposed e followed by a combining acute accent commonly forms one grapheme cluster and occupies three UTF-8 bytes. Treating its two code points as independent truncation positions can change the visible text even when both byte slices happen to decode.

These counts do not establish the size of a JSON body, an ASCII-only escape mode, a normalized value, a compressed message, or a framed transport. Measure those exact representations when the contract includes them.

Verify the procedure

For the concrete implementation, test and record:

• A value already within the byte and semantic limits.
• A value whose longest prefix lands exactly on the inclusive byte ceiling.
• A case where the next complete semantic unit exceeds the ceiling.
• Combining marks, emoji modifier sequences, joiner sequences, and regional-indicator pairs.
• Precomposed and decomposed text under the documented normalization policy.
• An omission marker that changes which prefix fits.
• A marker-only candidate exactly equal to the inclusive ceiling and a case where the marker alone is one byte too large.
• Quotes, backslashes, control characters, or other values whose serializer representation differs from the source string.
• Any pipeline feature that could make encoded size non-monotonic.

For a selected result, assert that the full production representation fits. Under a monotonic pipeline, also assert that adding the next complete semantic unit fails. Without a monotonicity proof, establish maximality by checking every longer permitted prefix or by another exhaustive equivalent.

Record actual runtime observations only after the tests execute. Until then, describe expected sizes and outcomes as reasoned examples.

Failure reporting

Report the original semantic measurement, selected semantic measurement, final encoded byte count, byte ceiling, counting unit, measured boundary, and whether a marker was inserted. Avoid logging the rejected text merely to explain its size.