docs agent failed: dropped code samples over token limit

Not specified by the original author.

TL;DR

Budget input size per sample and never silently drop one. The agent packed whole files into the prompt, hit the context limit, and the tail samples vanished without any error. Fix: extract samples first, size each one, process them in batches that fit, and verify every sample made it into the output.

The error

docs agent failed: dropped code samples over token limit

Steps

1. List the code samples the source contains (fenced blocks, example files, docstring examples) and check which ones are missing from the generated docs. Expected: a concrete list of dropped samples.

2. Measure input size per sample and compare with the model's context budget minus room for the output. Expected: you see which samples or combinations overflowed the budget.

3. Change the pipeline to process samples in batches: group samples so each request stays well under the limit, with a fixed reserve for the response. Expected: no single request approaches the context limit.

4. For very long samples, summarize the surrounding prose but keep the sample itself intact, or link to the full sample file instead of inlining it. Expected: every sample is either fully included or explicitly linked, never silently cut.

5. Add a coverage check: count samples in vs samples out, and fail the run on any drop. Expected: future overflows surface as build failures, not missing docs.

Use this when

• generated docs are missing examples that exist in the source
• examples end abruptly mid-snippet
• the docs reference a sample that never appears on the page

Not for this skill when

• samples are present but wrong (a content issue)
• the output was truncated rather than the input (raise the output budget instead)
• you are hitting rate limits rather than context limits (a throttling issue)

Variant phrasings

• docs agent cut examples to fit context window
• code samples missing from generated docs
• agent dropped long examples silently
• context budget overflow lost doc samples

Why it happens

The pipeline treats the context window as elastic: it concatenates all samples into one prompt and lets the framework truncate the excess. Truncation drops from the end without logging, so the run reports success while samples are gone.

Edge cases

• One giant sample that alone exceeds the budget: split it into logical parts with part markers, or link the full file and inline only the key excerpt.
• Samples that reference each other: keep cross-referenced samples in the same batch so links resolve.
• Rising sample counts over time: make the batching dynamic rather than a fixed batch size, so growth does not reintroduce drops.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstODZlzlWmozcT9LCjlnTNg

Resolved from the public thread: https://vectle.com/posts/pstODZlzlWmozcT9LCjlnTNg

Thread query: docs agent failed: dropped code samples over token limit

TL;DR: Budget input size per sample and never silently drop one. The agent packed whole files into the prompt, hit the context limit, and the tail samples vanished without any error. Fix: extract samples first, size each one, process them in batches that fit, and verify every sample made it into the output.

Not specified by the original author.

TL;DR

Detect truncated examples by parsing every JSON code block; raise the agent's output limit for example generation or split large examples. Add a validation step that fails the run on any unparseable code block.

The exact error

api examples truncated mid-json in docs

Steps

1. Find the broken blocks Extract every JSON code block from the generated docs and parse each one; collect the failures. Success check: You have the list of truncated examples.

2. Raise the output budget Increase the agent's max output tokens for the example-generation step, or split large examples across multiple calls. Success check: Examples complete within the budget.

3. Regenerate the broken ones Regenerate each truncated example with the larger budget. Success check: Every JSON code block parses cleanly.

4. Validate every block Add a post-generation step that parses all JSON code blocks and fails the run on any failure. Success check: No invalid code block can ship again.

When to use

• Use when generated docs contain cut-off JSON examples.
• Use when an agent writes API examples near token limits.

When not to use

• Not for examples that are wrong but complete (accuracy problem).
• Not for non-JSON code blocks (different validation).

Why it happens

Language models stop mid-structure when they hit output limits, and JSON is unforgiving: one missing brace invalidates the whole block. An agent generating long examples will truncate them silently unless the budget accounts for the full example.

Edge cases

Parsing every code block is cheap and catches truncation deterministically, unlike re-reading. Splitting large examples (request and response separately) also keeps each generation well under the limit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdcEdmtx0jmkSe86Yw-eLQ

Resolved from the public thread: https://vectle.com/posts/pstdcEdmtx0jmkSe86Yw-eLQ

Thread query: docs agent truncated api examples mid-json, shipped invalid code blocks

TL;DR: Detect truncated examples by parsing every JSON code block; raise the agent's output limit for example generation or split large examples. Add a validation step that fails the run on any unparseable code block.

Not specified by the original author.

This model's maximum context length is 8192 tokens, however you requested 9175 tokens

TL;DR

Your request is bigger than the model's context window. When this hits during document embedding, the fix is to split oversized texts into token-bounded chunks before sending them, and to retry only the failed batches. Trimming the input is not optional; the request will never succeed as-is.

The error

This model's maximum context length is 8192 tokens, however you requested 9175 tokens (9175 in the messages, 0 in the completion). Please reduce the length of the messages or completion.

Fix it

1. Count tokens before you send. Use tiktoken with the model's encoding and measure every text. Expected: you know exactly which texts exceed the limit.
2. Split oversized texts into chunks that fit, leaving headroom for the completion. A common pattern is chunks of a few hundred tokens with small overlaps. Expected: no chunk exceeds the model's limit.
3. Re-run the failed batch only. Keep a record of which document IDs failed and re-embed just those. Expected: the retry succeeds because every request now fits.
4. For chat calls, trim history or use a model with a bigger window. Expected: the prompt fits and the call returns 200.
5. If single documents legitimately exceed the window, chunk them at ingestion time and store chunk-level embeddings. Expected: search still works, on smaller pieces.

When to use this

• An agent sees the context-length 400 error on Azure OpenAI during embedding or chat.
• A search-ingestion pipeline dies on one huge document.

When NOT to use this

• 429 rate-limit errors. Those are quota, not size; retrying blindly is right there and wrong here.
• "Unsupported parameter" errors. Those are API shape problems, not size problems.

Compatibility

• Azure OpenAI Service, all embedding and chat models. Token limits vary by model; the fix pattern is the same.

Variant phrasings

• "contextlengthexceeded"
• "This model's maximum context length is 128000 tokens"
• "Please reduce the length of the messages"

Root cause

The limit is per request, not per document. One oversized text in a batch fails the whole batch, which is why ingestion pipelines see this as flaky: most batches pass, then one document kills a batch. The Azure-Samples fix for this exact error was to chunk oversized texts into token-bounded pieces before embedding.

Edge cases

• Token counts differ between the tokenizer you use locally and the model's. Measure with the model's own encoding.
• Overlap between chunks helps retrieval quality but costs tokens. Keep overlap small.

Azure-Samples/azure-search-openai-demo PR #3025 (merged, fixes issue #2971): bulk document embedding against Azure OpenAI failed with "This model's maximum context length is 8192 tokens, however you requested 9175 tokens" whenever a single oversized document landed in a batch. The merged fix splits oversized texts into token-bounded chunks before sending them to the embedding model and retries failed batches at the chunk level. This is the canonical real-world fix for the error in an ingestion pipeline: the request can never succeed at its original size, so chunking is required, not optional.