agent's sphinx directive parser failed on custom extensions

TL;DR

Teach the agent to stop validating directives it does not know: either parse with the project's Sphinx extensions loaded, or treat unknown directives as opaque blocks (directive line plus indented body, no argument checking). This fixes the failure because directive syntax (allowed arguments, options, body shape) is defined by each directive's class, not by generic RST rules, so a generic parser can never validate custom ones correctly.

The error

agent's sphinx directive parser failed on custom extensions

Steps to fix

1. Reproduce the failure: run the agent's parser over the failing .rst file and record the exact directive name it chokes on (the line starting with .. followed by the name and ::).

   • Expected: one or two directive names, not a broad parse failure across the whole tree.

2. Open the project's conf.py and find which extension registers that directive (search for add_directive calls or the extension module name in the extensions list).

   • Expected: the directive maps to a project extension, confirming the parser just doesn't know it.

3. Apply the fix. Option A (best): parse through Sphinx's own machinery with the project's conf.py loaded so every custom directive is registered before parsing. Option B (simpler): add the custom directive names to the agent parser's known-directive table as opaque blocks - consume the directive line and its indented body verbatim without validating arguments or options.

   • Expected: the parser accepts the custom directive lines without error.

4. Re-run the parser over the entire docs tree and diff the extracted text against the previous run.

   • Expected: zero parse errors, and the custom directive content is preserved in the output.

Use this when

• A docs agent parses .rst files with its own parser and fails only on files using custom directives.
• The failing directive names are registered by extensions in the project's conf.py.
• The agent needs directive text extracted, not validated (summaries, search indexing, doc sync).

Not for this skill when

• Sphinx itself fails to build (that's a build config problem, not an agent parser problem).
• The failure is on standard directives like note or code-block (check the parser's built-in directive table instead).
• The agent generates directives rather than parsing them.

Variant phrasings

• docs agent can't parse custom sphinx directives
• sphinx directive parser error on project-specific extensions
• agent rst parser fails on unknown directive
• custom admonition directive breaks docs agent parsing

Why it happens

RST directives are an extension point: .. name:: can take any arguments, options, and body shape depending on the registered directive class. A standalone parser only knows the built-in directive set, so when it meets a custom one it either rejects the unknown name or misparses its arguments. The parser was written against the generic RST grammar, but custom directives don't follow a generic grammar - they follow whatever the extension author defined.

Edge cases

• Directives registered via plain docutils (not Sphinx) in a standalone module: conf.py won't list them, so scan for directives modules in the repo too.
• Nested directives inside a custom directive's body: the opaque-block approach must still recurse into the body for nested standard directives.
• Custom roles (single-backtick roles) fail the same way: apply the same known-or-opaque treatment to roles.
• Option parsing differences: never try to validate options on an opaque directive, just carry them through verbatim.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstSd4douBL5DT6yetNiVBmw