docs agent failed: copied examples that do not compile

TL;DR

Compile or run every copied example before it ships in the docs. The agent pasted snippets from issues, old READMEs, or chat output as plain text, so wrong imports and renamed functions went unnoticed. Fix: extract each code block, run it through the real compiler or interpreter, fix failures, and publish only when every block exits clean.

The error

docs agent failed: copied examples that do not compile

Steps

1. Extract every fenced code block from the changed docs files into temp files, one file per block, named by language. Expected: a directory of snippet files with one file per example.

2. Run the right check per language: python -m py_compile for Python, node --check for JavaScript, javac for Java, go build for Go. Expected: exit 0 on every snippet. A nonzero exit is a broken example.

3. Read the first compiler error for each failure. The usual causes are stale import paths, renamed functions, or missing dependency installs. Expected: you can point at the exact line and the exact symbol that is wrong.

4. Fix the snippet to match the current API, and add any missing setup line the example assumed (install command, env var). Expected: the check from step 2 now exits 0.

5. Gate the docs job on this check so the agent cannot publish without it. Expected: docs CI runs the compile check and blocks merge on failure.

Use this when

• doc examples fail to compile or run for users
• the agent copied snippets from issues, PRs, or old docs without testing
• users report "the example in your docs does not work"
• docs were generated by an agent with no execution step in the pipeline

Not for this skill when

• examples compile fine but do the wrong thing (logic bug, different skill)
• the docs contain prose only, no code
• the failure is in application code, not doc examples

Variant phrasings

• example code in docs fails to build
• documentation snippet throws ImportError or SyntaxError on copy
• agent wrote docs with broken sample code
• readme example does not run on a fresh checkout

Why it happens

The agent treats example code as text, not code. An LLM reproduces plausible-looking snippets from its training data or from other files in the repo, and nothing in a naive pipeline executes them, so stale imports, renamed APIs, and missing setup steps slip through into published docs.

Edge cases

• Examples needing credentials or network access: compile-check them, but stub the network calls or skip execution; note required setup in the docs.
• Multi-file examples: check them together in dependency order, not one block at a time.
• Version-specific snippets: pin the language version in the check and state it above the example.
• Interactive REPL sessions: strip the REPL prompts before compiling, or keep a canonical script version.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstLGhobU3q8gwYmZKnsu--w