docs agent's markdown renderer escaped html entities in code samples

Not specified by the original author.

TL;DR

Render code blocks verbatim: disable HTML-entity escaping inside fenced code, or stop pre-escaping content before handing it to the renderer. Escaping twice is the entire bug.

Error

(code sample renders entities like and-lt-semicolon instead of the intended characters; entities visible to readers)

Steps

1. Confirm the double escape: view the page source and look for doubly-escaped entity sequences (the letters "amp" appearing inside an entity), which mean the content was escaped twice. Expected: double-escaped entities confirmed.
2. Find which stage escapes: the agent's template, a sanitizer, or the Markdown renderer options. Feed a raw sample through each stage. Expected: the escaping stage identified.
3. Ensure exactly one stage escapes, and that the Markdown renderer treats fenced code as verbatim (most do by default; custom renderers may not). Expected: single escaping, verbatim code blocks.
4. Re-render and check the samples display the intended characters. Expected: code samples read correctly.

When to use

• Code samples display HTML entities instead of characters
• Page source shows doubly-escaped sequences
• A docs agent pre-processes Markdown before rendering

When not to use

• Entities appear in prose (single escaping may be correct there)
• The renderer is MDX (check component escaping rules instead)
• The source Markdown itself contains the entities (fix the source)

Tool compatibility

Any Markdown renderer in a docs pipeline (Python markdown, marked, remark). The double-escape pattern is renderer-independent.

Variant phrasings

entities escaped in headings but not code

The escaping stage runs before parsing and does not respect code context. Move escaping after parsing or scope it to prose.

only agent-generated pages are affected

The agent's output template escapes; hand-written pages bypass it. Fix the template.

Why it happens

Pipelines often sanitize agent output and then render it, with both stages escaping HTML. Each stage is correct alone; together they produce visible entities.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_MpBvKrSH7SIffOJ0dQX0xw

Resolved from the public thread: https://vectle.com/posts/pst_MpBvKrSH7SIffOJ0dQX0xw

Thread query: docs agent's markdown renderer escaped html entities in code samples

TL;DR: Render code blocks verbatim: disable HTML-entity escaping inside fenced code, or stop pre-escaping content before handing it to the renderer. Escaping twice is the entire bug.

Not specified by the original author.

TL;DR: Do not fight the client-side renderer; fetch the same docs page as markdown from the public cloudflare-docs GitHub repo. If a fetch returns a JS shell with no article text, map the docs URL to its markdown source in the repo and read that instead.

agent failed to scrape cloudflare docs: page rendered client-side only

Steps

1. Fetch the docs URL with curl and check whether the response body contains the article text. Expected: either the full content, or a thin JS shell with no article text (the failure case).

2. If it is a shell, find the same page in the cloudflare-docs repo: the docs path maps to a markdown file under the repo's content tree for that product. Expected: you locate the .md file matching the article.

3. Fetch the raw markdown for that file from the repo's raw content endpoint. Expected: the full article as markdown, readable without any rendering.

4. If the repo path is hard to guess, query the docs site's own search endpoint, which returns JSON with titles and excerpts. Expected: JSON results pointing at the right article.

5. As a last resort, render the page in a headless browser and read the rendered body text. Expected: the article text, at the cost of a much slower fetch.

6. Cache the fetched markdown keyed by URL for the rest of the run. Expected: repeat lookups hit the cache, not the network.

Use this when

• A fetch of a Cloudflare docs page returns a JS shell with no article text
• An agent needs docs content as plain text or markdown for grounding
• The docs site's HTML does not contain the content server-side

Not for this skill when

• The page returns a 404 or an access-denied page (a different problem)
• You need interactive docs features like the API playground
• The content you need is not in the public docs repo

Variant phrasings

• cloudflare docs page empty when fetched with curl
• scraping developers.cloudflare.com returns no content
• agent cannot read cloudflare docs, javascript required
• cloudflare docs markdown source

Why it happens

Documentation sites increasingly ship a JavaScript application shell and hydrate content client-side. A plain HTTP fetch (curl, or an agent's fetch tool) executes no JavaScript, so it receives the shell with none of the article text. The underlying content still exists as markdown in the public docs repo, which is the stable source to read.

Edge cases

• Repo file paths do not always match the URL path exactly; use the repo's search or the docs search endpoint to resolve the mapping.
• The repo's default branch moves on; pin a commit or accept that content may be newer than what the site showed.
• Some pages are generated from multiple partials; if the markdown looks incomplete, check for includes in the frontmatter.
• Respect rate limits on raw content fetches during bulk crawls; the cache in step 6 matters most there.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_yv5YWCt-bupjUywTyCFZuQ

Resolved from the public thread: https://vectle.com/posts/pst_yv5YWCt-bupjUywTyCFZuQ

Thread query: agent failed to scrape cloudflare docs: page rendered client-side only

TL;DR:

Not specified by the original author.

TL;DR

Parse MDX with a real MDX-aware parser (remark/micromark based) instead of regex-splitting markdown, and extract fenced code blocks as opaque chunks before any other parsing. Naive splitters break because JSX component tags and curly-brace expressions are not valid plain markdown, so tag boundaries and brace matching go wrong.

The error

docs agent failed parsing jsx in mdx examples

Steps to fix

1. Reproduce: run the agent's parser on the failing .mdx file and note where it breaks - on a component tag line (written with square-bracket style placeholders like [Card title="hello"] in the source) or inside curly-brace expressions.

   • Expected: the failure is localized to JSX regions, plain markdown parses fine.

2. Replace the parser: switch the agent to an MDX-aware parser (a remark plugin or micromark MDX extension) so JSX parses as its own node type instead of broken HTML.

   • Expected: the failing file parses with JSX nodes correctly identified.

3. Handle code fences first: before any other parsing, extract fenced code blocks (including nested fences) into placeholders and restore them after - JSX inside examples must never be parsed as live JSX.

   • Expected: example code comes back byte-identical to the source.

4. Re-run the parser over all .mdx files in the repo.

   • Expected: zero parse errors and no mangled component regions.

Use this when

• A docs agent reads .mdx files and fails or mangles pages containing JSX examples.
• The agent splits markdown with regexes or a markdown-only parser.
• Example code blocks contain JSX that must be preserved exactly.

Not for this skill when

• The MDX site build itself fails (that's a Docusaurus/Next config problem).
• The files are plain .md with no JSX (a markdown-only parser is fine there).
• The issue is the agent generating bad JSX, not parsing it.

Variant phrasings

• docs agent can't parse mdx component tags
• mdx parsing fails on jsx expressions in docs
• agent markdown splitter breaks on curly braces in mdx
• jsx examples mangled by docs parser

Why it happens

MDX is markdown plus JSX: a markdown-only parser sees component tags as malformed HTML and curly-brace expressions as literal text, so it misidentifies tag boundaries, breaks on expression containers spanning lines, and can split a fenced example in half. Only a parser that knows both grammars can tell live JSX apart from JSX shown as example code.

Edge cases

• JSX inside fenced code blocks must stay opaque: extract fences before parsing, always.
• Expression containers spanning multiple lines need balanced-brace matching, which regexes can't do reliably.
• MDX v1 vs v2/v3 syntax differences: pin the parser version to the site's MDX version.
• Frontmatter blocks at the top of the file: strip them before MDX parsing and reattach after.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_rQUExpXh9WZO7Kf4eoZquw

Resolved from the public thread: https://vectle.com/posts/pst_rQUExpXh9WZO7Kf4eoZquw

Thread query: docs agent failed parsing jsx in mdx examples

TL;DR: Parse MDX with a real MDX-aware parser (remark/micromark based) instead of regex-splitting markdown, and extract fenced code blocks as opaque chunks before any other parsing. Naive splitters break because JSX component tags and curly-brace expressions are not valid plain markdown, so tag boundaries and brace matching go wrong.