docs agent failed: sphinx build crashed on non-UTF8 docstring during repo scan

TL;DR

Detect the offending file from the traceback, convert it to UTF-8, and add an encoding check to the agent's scan so future bad files are quarantined instead of crashing the build. Log the file and continue the scan.

The exact error

UnicodeDecodeError during sphinx build on agent-scanned repo

Steps

1. Find the offending file Read the traceback for the file path the agent was processing when decoding failed. Success check: You have the exact file with the bad encoding.

2. Detect and convert the encoding Run file or chardet on it to identify the encoding, then convert to UTF-8. Verify the docstrings still read correctly. Success check: The file is valid UTF-8 and its content is unchanged.

3. Add an encoding guard In the agent's scan loop, try decoding each file as UTF-8 first; on failure, log the file, skip it, and continue instead of crashing. Success check: One bad file produces a warning, not a crashed run.

4. Rerun the full scan Run the agent's docs generation end to end and confirm it completes with the guard in place. Success check: The scan completes and the build succeeds.

When to use

• Use when a docs agent's Sphinx build crashes on file encoding.
• Use when scanning legacy repos with mixed encodings.

When not to use

• Not for UTF-8 files with genuinely invalid syntax (different error).
• Not for encoding issues in the built HTML output (output encoding config).

Why it happens

Sphinx assumes UTF-8 source files. Repos with legacy latin-1 or windows-1252 files decode-fail the moment autodoc or the parser touches them, and an agent scanning the whole repo hits the bad file unattended.

Edge cases

The guard belongs in the scan loop, not in Sphinx config, because the agent decides which files enter the build. Converting the file is a repo fix; the guard is the agent fix. Both are needed.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_uJOlqjbThm9S0RraNaJxwQ