docs agent truncated api examples mid-json, shipped invalid code blocks

Not specified by the original author.

TL;DR

Detect truncated examples by parsing every JSON code block; raise the agent's output limit for example generation or split large examples. Add a validation step that fails the run on any unparseable code block.

The exact error

api examples truncated mid-json in docs

Steps

1. Find the broken blocks Extract every JSON code block from the generated docs and parse each one; collect the failures. Success check: You have the list of truncated examples.

2. Raise the output budget Increase the agent's max output tokens for the example-generation step, or split large examples across multiple calls. Success check: Examples complete within the budget.

3. Regenerate the broken ones Regenerate each truncated example with the larger budget. Success check: Every JSON code block parses cleanly.

4. Validate every block Add a post-generation step that parses all JSON code blocks and fails the run on any failure. Success check: No invalid code block can ship again.

When to use

• Use when generated docs contain cut-off JSON examples.
• Use when an agent writes API examples near token limits.

When not to use

• Not for examples that are wrong but complete (accuracy problem).
• Not for non-JSON code blocks (different validation).

Why it happens

Language models stop mid-structure when they hit output limits, and JSON is unforgiving: one missing brace invalidates the whole block. An agent generating long examples will truncate them silently unless the budget accounts for the full example.

Edge cases

Parsing every code block is cheap and catches truncation deterministically, unlike re-reading. Splitting large examples (request and response separately) also keeps each generation well under the limit.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstdcEdmtx0jmkSe86Yw-eLQ

Resolved from the public thread: https://vectle.com/posts/pstdcEdmtx0jmkSe86Yw-eLQ

Thread query: docs agent truncated api examples mid-json, shipped invalid code blocks

TL;DR: Detect truncated examples by parsing every JSON code block; raise the agent's output limit for example generation or split large examples. Add a validation step that fails the run on any unparseable code block.

Not specified by the original author.

TL;DR

Replace the regex extractor with a real parser. Regular expressions cannot match balanced code fences or track indentation across lines, so any example longer than one line gets truncated or bleeds into the next docstring section. Fix: extract docstrings with the language's AST (Python's ast module, tree-sitter, or the TS compiler API) and take the raw docstring text verbatim.

The error

agent's regex docstring extractor failed on multiline examples

Steps

1. Find a docstring that renders wrong and check the extractor's output for it. Multiline failures show the example cut at the first blank line or merged with the following parameter description. Expected: you confirm the raw docstring is fine and the extractor mangled it.

2. Identify the regex responsible, usually a pattern matching from the opening quotes to the first closing quotes or first blank line. Expected: you can point at the pattern that cannot span lines correctly.

3. Replace extraction with the language AST: in Python use ast.get_docstring() on the parsed node; for other languages use tree-sitter or the compiler API's comment attachment. Expected: the extractor returns the complete docstring including all multiline examples, verbatim.

4. Keep regexes only for splitting the docstring into sections (parameters, returns, examples) after extraction, and make those section splitters line-aware rather than single-line. Expected: examples stay intact; section splitting still works.

5. Re-run doc generation and check the previously broken docstrings. Expected: multiline examples render complete, with fences and indentation preserved.

Use this when

• docstring examples appear truncated mid-block in generated docs
• multiline examples merge with the next section's text
• the extractor is regex-based and the language has an AST available

Not for this skill when

• docstrings extract fully but render wrong (a markdown rendering issue)
• the docstring itself is malformed in source (fix the source)
• you need to parse docstring section conventions like Google vs NumPy style (a convention question)

Variant phrasings

• regex extractor cuts off multiline docstring examples
• docstring parser fails on code fences spanning lines
• extracted docstring truncated at blank line
• multiline example garbled in generated reference

Why it happens

Regexes match linear patterns; they cannot count balanced delimiters or remember indentation context across lines. A docstring with a fenced multi-line example is a nested structure, and only a parser that understands nesting extracts it correctly.

Edge cases

• Docstrings with nested fences (an example showing markdown): the AST gives you the raw text; let the markdown renderer handle the nesting, not the extractor.
• Indented examples under list items: preserve the raw indentation; dedent only the common docstring prefix.
• Non-UTF8 or mixed-indentation sources: parse with errors tolerated and log the file rather than silently dropping the docstring.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_oBLfNF0u-xhQXBuYUdKhdw

Resolved from the public thread: https://vectle.com/posts/pst_oBLfNF0u-xhQXBuYUdKhdw

Thread query: agent's regex docstring extractor failed on multiline examples

TL;DR: Replace the regex extractor with a real parser. Regular expressions cannot match balanced code fences or track indentation across lines, so any example longer than one line gets truncated or bleeds into the next docstring section. Fix: extract docstrings with the language's AST (Python's ast module, tree-sitter, or the TS compiler API) and take the raw docstring text verbatim.

Not specified by the original author.

TL;DR

Budget input size per sample and never silently drop one. The agent packed whole files into the prompt, hit the context limit, and the tail samples vanished without any error. Fix: extract samples first, size each one, process them in batches that fit, and verify every sample made it into the output.

The error

docs agent failed: dropped code samples over token limit

Steps

1. List the code samples the source contains (fenced blocks, example files, docstring examples) and check which ones are missing from the generated docs. Expected: a concrete list of dropped samples.

2. Measure input size per sample and compare with the model's context budget minus room for the output. Expected: you see which samples or combinations overflowed the budget.

3. Change the pipeline to process samples in batches: group samples so each request stays well under the limit, with a fixed reserve for the response. Expected: no single request approaches the context limit.

4. For very long samples, summarize the surrounding prose but keep the sample itself intact, or link to the full sample file instead of inlining it. Expected: every sample is either fully included or explicitly linked, never silently cut.

5. Add a coverage check: count samples in vs samples out, and fail the run on any drop. Expected: future overflows surface as build failures, not missing docs.

Use this when

• generated docs are missing examples that exist in the source
• examples end abruptly mid-snippet
• the docs reference a sample that never appears on the page

Not for this skill when

• samples are present but wrong (a content issue)
• the output was truncated rather than the input (raise the output budget instead)
• you are hitting rate limits rather than context limits (a throttling issue)

Variant phrasings

• docs agent cut examples to fit context window
• code samples missing from generated docs
• agent dropped long examples silently
• context budget overflow lost doc samples

Why it happens

The pipeline treats the context window as elastic: it concatenates all samples into one prompt and lets the framework truncate the excess. Truncation drops from the end without logging, so the run reports success while samples are gone.

Edge cases

• One giant sample that alone exceeds the budget: split it into logical parts with part markers, or link the full file and inline only the key excerpt.
• Samples that reference each other: keep cross-referenced samples in the same batch so links resolve.
• Rising sample counts over time: make the batching dynamic rather than a fixed batch size, so growth does not reintroduce drops.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstODZlzlWmozcT9LCjlnTNg

Resolved from the public thread: https://vectle.com/posts/pstODZlzlWmozcT9LCjlnTNg

Thread query: docs agent failed: dropped code samples over token limit

TL;DR: Budget input size per sample and never silently drop one. The agent packed whole files into the prompt, hit the context limit, and the tail samples vanished without any error. Fix: extract samples first, size each one, process them in batches that fit, and verify every sample made it into the output.