Partition Unicode Records Into Byte-Capped Serialized Batches

Use this procedure when an API accepts an ordered batch of records, each text field has a semantic Unicode limit, and every serialized request has an independent UTF-8 byte ceiling. Records must remain whole: the batching layer may move a record to the next request but may not truncate or split it.

This is narrower than validating one text value. A batch can fail its byte ceiling even when every field passes its character limit, because the wire body also contains property names, quotes, escapes, commas, brackets, and other envelope data.

Freeze both contracts

Record these facts before batching:

• The semantic counting unit for each limited field: code points, grapheme clusters, or a contractually named code-unit scheme.
• Any normalization required before semantic counting.
• The exact byte-limited representation: the serialized body, a framed message, compressed bytes, or another named boundary.
• The production serializer settings, including Unicode escaping, whitespace, property ordering when relevant, and optional-field behavior.
• Whether the byte ceiling is inclusive.
• Whether input order must be preserved.
• The required action when one otherwise valid record cannot fit in an empty batch.

Do not convert a byte ceiling into a character estimate. Do not add the encoded sizes of source strings and call that the request size.

Validate records before packing

For every record:

1. Validate each limited field with its specified semantic unit and normalization policy.
2. Keep semantic failures distinct from batch-size failures.
3. Apply all production transformations that are part of the record before serialization.
4. Treat the resulting record as indivisible unless the API contract explicitly permits record splitting.

A record that exceeds a field character limit is invalid regardless of how many request bytes remain. A record whose fields pass can still be too large for any request.

Measure complete candidate batches

Define one production encoder that accepts a candidate record list and returns the exact bytes that would be sent at the contracted boundary.

For every candidate, include:

• The complete top-level envelope.
• Keys, brackets, commas, quotes, and whitespace emitted by production settings.
• Escaping chosen by the real serializer.
• Optional fields and metadata whose presence depends on batch contents.
• Framing or compression only when the byte contract applies after that stage.

Cache or retain the accepted encoded bytes and send those exact bytes. Re-serializing later can invalidate the measurement if configuration, optional data, timestamps, or ordering can change.

Pack an ordered batch safely

When adding records in order:

1. Start with an empty candidate batch.
2. Append the next whole record to a candidate copy.
3. Encode the complete candidate with the production encoder.
4. If its byte count is within the inclusive ceiling, accept the candidate and retain its encoded bytes.
5. If it exceeds the ceiling and the current batch is nonempty, emit the previously accepted bytes. Start a new candidate containing the same record; do not skip it.
6. Measure that singleton candidate.
7. If the singleton fits, continue from it.
8. If the singleton exceeds the ceiling, report an oversized-record failure using its position or safe identifier. Do not retry it forever, truncate it implicitly, or mislabel it as a character-limit failure.

This greedy longest-prefix procedure is valid when adding a record cannot reduce the measured size. That property normally holds for an ordinary uncompressed deterministic array encoding, but it must be proved for the actual boundary. Compression, deduplication, conditional envelope fields, and content-dependent framing can make size non-monotonic. When monotonicity is not established, measure the allowed groupings explicitly or change the contract; do not infer a safe split from a failed larger candidate.

Keep failure identities separate

Report at least:

• Semantic field limit exceeded.
• Record cannot fit in an otherwise empty byte-capped request.
• Candidate batch exceeds the byte ceiling and was split normally.
• Serialization or encoding failed.

For observability, record the semantic unit, relevant limit, measured byte boundary, final byte count, and record count. Avoid logging rejected text merely to explain its size.

Reasoned example

The following is a reasoned UTF-8 calculation, not an executed test.

Assume:

• Each value has an inclusive one-code-point limit.
• The batch body is a compact JSON array.
• The serializer emits the grinning-face character directly rather than using an ASCII escape.
• The inclusive request ceiling applies to the UTF-8 bytes of the complete array.

The compact body containing records with values ASCII A and grinning face is:

[{"s":"A"},{"s":"😀"}]

Both values contain one code point, so both pass the semantic limit. The complete body occupies twenty-four UTF-8 bytes: one byte for the opening bracket, nine for the first object, one for the comma, twelve for the second object, and one for the closing bracket.

Under a twenty-four-byte ceiling, the two-record batch fits exactly. Under a twenty-three-byte ceiling, it does not, even though both character checks pass. The separate singleton bodies occupy eleven and fourteen bytes, so the records can be sent as two ordered batches.

If the production serializer instead emits the grinning face as the twelve ASCII characters of a JSON surrogate-pair escape, the two-record body occupies thirty-two bytes. That difference is why the procedure measures the production serialization rather than estimating from decoded character counts.

Verification checklist

Execute implementation tests before claiming runtime evidence:

• Every field passes and the complete batch lands exactly on the byte ceiling.
• Every field passes but envelope and separator overhead push the batch one byte over.
• A multibyte value passes its semantic limit while causing a batch split.
• A field fails its semantic limit even though its singleton request would fit.
• The first record cannot fit in an empty batch and produces one terminal oversized-record result.
• The last accepted encoded bytes are exactly the bytes handed to the transport.
• Serializer modes that emit direct UTF-8 and ASCII escapes produce their independently expected decisions.
• Empty input and a one-record input include the correct top-level envelope cost.
• If greedy packing is used, tests establish the required monotonicity for the measured representation.

Until those tests are actually run, describe expected sizes and decisions only as reasoned examples.