Prove Independent Unicode and UTF-8 Limits with a Crossed Test Matrix

TL;DR

Use this procedure when one network field or payload must satisfy both a semantic Unicode length limit and a UTF-8 byte ceiling.

When to use

You are seeing this: It is especially useful when tests currently cover only ASCII or only one rejection path. Use this skill when you run into "Prove Independent Unicode and UTF-8 Limits with a Crossed Test Matrix".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Use this procedure when one network field or payload must satisfy both a semantic Unicode length limit and a UTF-8 byte ceiling. It is especially useful when tests currently cover only ASCII or only one rejection path.

The objective is not merely to test each limit once. It is to prove that the implementation computes two different measurements, applies both decisions, and attributes rejection to the correct constraint.

Freeze the two contracts

Before choosing fixtures, write down:

• The semantic unit: grapheme clusters, Unicode code points, or language-specific code units.
• The exact value to which the semantic limit applies.
• The byte boundary: final field value, serialized body, framed message, or another explicitly named representation.
• The encoding and serializer settings at that boundary.
• Required normalization and whether it occurs before either measurement.
• Whether each maximum is inclusive.
• Whether validation reports every violated constraint or stops at the first one.

Do not build the matrix until these details are fixed. A fixture can move between cells if normalization, escaping, framing, or the semantic counting unit changes.

Construct all four outcomes

Choose fixtures that produce this matrix:

The two disagreement rows are the essential part. If they cannot be constructed for the chosen ceilings, use test-only ceilings that preserve the production counting rules, or test the validators separately as well as through the combined entry point. Do not weaken production semantics merely to make a fixture convenient.

Measure the semantic value with the declared Unicode unit. Produce the byte value by running the scoped representation through the production normalization, serialization, and UTF-8 encoding path, then count the resulting bytes. Do not derive both expected values from the same production helper being tested.

Reasoned example

Assume an inclusive maximum of two Unicode code points and an inclusive maximum of four UTF-8 bytes. The byte ceiling applies to the final field value, with ordinary UTF-8 encoding and no normalization or escaping after validation.

These are reasoned encoding examples, not executed tests. They do not establish counts for a serialized body, an ASCII-only serializer, a normalized value, a framed message, or a different Unicode counting unit.

For a grapheme-cluster limit, replace the semantic expectations with grapheme segmentation results and retain an independently measured byte oracle. For a code-unit contract, count exactly the code units named by that contract even when they differ from code points.

Exercise exact boundaries

For each validator, add a fixture exactly at its inclusive ceiling and one just beyond it while holding the other dimension on the intended side of its limit. This prevents an off-by-one result from being hidden by the other validator.

When possible, include:

• ASCII text, which can make the two counts accidentally look interchangeable.
• Two-byte, three-byte, and four-byte UTF-8 scalar values.
• Precomposed and decomposed forms when the contract does not normalize them to the same representation.
• Multi-code-point grapheme clusters when the semantic unit is user-perceived text.
• Characters whose serializer escaping changes the measured body size when the byte limit applies after serialization.
• Fixed envelope overhead when the ceiling applies to the whole body.

Assert the decision and its attribution

For every fixture, assert more than a generic rejection:

• The semantic measurement and its named unit.
• The encoded byte measurement and its named boundary.
• The final accept or reject decision.
• The specific violated constraint or constraints.
• The documented precedence when both fail and the API returns only one error.

A combined validator may short-circuit for efficiency, but its tests must still prove the skipped path independently. Otherwise the both-fail row can pass while one validator is absent.

Avoid returning or logging the entire rejected payload merely to explain the size error. Counts, limits, units, boundary names, and safe field identifiers are usually sufficient.

Detect common false confidence

This matrix is designed to catch these defects:

• A string-length helper is reused as a UTF-8 byte count.
• Byte length is measured before required normalization or serialization.
• A field-level measurement is substituted for a whole-body limit.
• ASCII-only fixtures make character count and byte count appear equivalent.
• The first validator always rejects, leaving the second path untested.
• The final decision is correct but the reported constraint is wrong.
• Expected counts are computed with the same faulty helper as the production result.

Verification record

When the tests are actually run, record the runtime, serializer mode, normalization policy, measured representation, and observed counts. Label those observations as executed results.

Until that execution occurs, describe the fixture tables only as reasoned examples. Do not present inferred counts as runtime evidence.

The procedure was derived by comparing existing public guidance on independent Unicode and UTF-8 limits with the narrower need to prove validator independence. The concrete counts in the example are reasoned from ordinary UTF-8 encoding for a final field value with no normalization or escaping after validation. No executable test, packet capture, runtime measurement, external source, or conversation evidence was used.

Rationale: Existing guidance explains that semantic counts and UTF-8 bytes are separate invariants and includes a semantic-pass, byte-fail example. A narrower missing procedure is to force both disagreement directions plus the two agreement cases in one test matrix. That matrix can reveal implementations that reuse character length as byte length, never reach the second validator, or report the wrong violated constraint.