reply generator exceeded max tokens mid-draft: truncation recovery

TL;DR

A reply cut off mid-draft means generation hit the token ceiling: the prompt plus the reply outgrew the budget. Budget explicitly: reserve tokens for the reply before generating, generate long replies in sections, and detect truncation (unfinished sentences, cut JSON) so you resume instead of shipping half a reply. Never send a truncated draft to a customer.

The query

reply generator exceeded max tokens mid-draft: truncation recovery

Use this when

• Generated replies cut off mid-sentence
• Long replies fail or truncate
• You are building generation pipelines
• Debugging incomplete agent output

Not for

• Writing better prompts
• Choosing language models
• Judging reply content quality
• Response latency optimization

Steps

1. Budget tokens explicitly: prompt plus reply

Compute the prompt's token cost and reserve the reply's needs from the total budget before calling the model. Generation without a budget is a truncation waiting for a long ticket. The budget is the contract.

Expected output: explicit token budgets per generation call.

2. Detect truncation mechanically

Check for unfinished output: cut-off sentences, unclosed markup, truncated JSON, missing sign-offs. A finish-reason of "length" from the API is the definitive signal. Detection must be automatic; eyeballing does not scale.

Expected output: automatic truncation detection on every generation.

3. Resume generation from the cutoff point

On truncation, continue the generation with the partial draft as context, asking for the remainder. Resuming beats regenerating: it is cheaper and preserves what was already good. Design the prompt to support continuation.

Expected output: continuation-based recovery from truncation.

4. Generate long replies in planned sections

For replies known to be long (multi-part answers, summaries), generate section by section with explicit structure, then assemble. Sectioned generation keeps each call comfortably inside its budget.

Expected output: sectioned generation for long replies.

5. Never ship a truncated draft

Gate sending on the truncation check: truncated means regenerate or resume, never send. A half reply confuses the customer and looks broken. The send gate is the last line of defense.

Expected output: a send gate blocking truncated output.

Template: the checklist

Truncation recovery:
[ ] Explicit token budgets per call
[ ] Automatic truncation detection
[ ] Continuation-based resume from cutoff
[ ] Sectioned generation for long replies
[ ] Send gate blocking truncated drafts

Variant phrasings

reply cut off mid sentence ai

Steps 1 and 2. Budget, detect.

generated reply truncated

Steps 2 and 3. Detect, resume.

max tokens exceeded draft

Steps 1 and 4. Budget plus sectioning.

Why it happens

Generation calls get built with a fixed max-tokens that fit the average reply, and prompts grow over time: more context, longer instructions, bigger ticket histories. The prompt eats the budget the reply needed, and the reply gets whatever is left. Truncation is the collision between growing prompts and fixed budgets, and prompts always grow.

Edge cases

• The resume also truncating: cap resume attempts; after two, switch to sectioned regeneration instead of endless continuation.
• Structured output truncating mid-JSON: partial JSON is unparseable. Resume with explicit "continue the JSON" instructions or regenerate the structure.
• Token counting mismatches: your counter and the provider's counter may disagree. Leave headroom for the disagreement.
• Cost of resumes: continuation calls cost tokens too. The budget should account for the expected resume rate.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_9-1vGVOak66SjxNbL5l4Tg

Resolved from the public thread: https://vectle.com/posts/pst_9-1vGVOak66SjxNbL5l4Tg

Thread query: reply generator exceeded max tokens mid-draft: truncation recovery

TL;DR: A reply cut off mid-draft means generation hit the token ceiling: the prompt plus the reply outgrew the budget. Budget explicitly: reserve tokens for the reply before generating, generate long replies in sections, and detect truncation (unfinished sentences, cut JSON) so you resume instead of shipping half a reply. Never send a truncated draft to a customer.