nbconvert failed on notebook with non-utf8 output bytes, UnicodeDecodeError

TL;DR

A cell output with bytes that are not valid UTF-8 (latin-1 text, binary blobs printed as text) makes nbconvert's writer fail when it serializes the notebook. Find the offending cell by converting with output capture and reading the traceback's cell reference, then either re-encode the data properly in the cell or clear that cell's outputs. The notebook runs fine interactively because the browser is lenient; nbconvert is not.

nbconvert failed on notebook with non-utf8 output bytes, UnicodeDecodeError

Use this when

• nbconvert fails with UnicodeDecodeError while the notebook runs fine.
• A cell prints binary data, latin-1 text, or subprocess output with odd bytes.
• The notebook was generated on a machine with a non-UTF-8 locale.

Not for this skill when

• The .ipynb file itself will not parse. That is a file encoding issue.
• Python code raises UnicodeDecodeError. That is a code bug, not nbconvert.

Steps

1. Get the failing cell: run the export and read the traceback for the cell index. Verify: you know which cell owns the bad bytes.
2. Inspect that cell's outputs in the notebook JSON for non-UTF-8 sequences. Verify: you can see the offending output.
3. Fix at the source: decode the bytes with the right codec in the cell (e.g. latin-1) or stop printing binary data. Verify: the cell's outputs are clean text.
4. Alternatively, clear just that cell's outputs before export. Verify: the export proceeds past the cell.
5. Re-run the export. Verify: it completes without UnicodeDecodeError.

Variant phrasings

nbconvert unicodedecodeerror

Short form. A cell output has bad bytes.

nbconvert non-utf8 output

The cause phrasing. Re-encode or strip.

unicode error exporting notebook

General phrasing. The browser tolerates what nbconvert rejects.

Compatibility: nbconvert 6/7, Python 3. The writer assumes UTF-8 throughout.

Why it happens

The notebook format mandates UTF-8, but kernels can stuff arbitrary bytes into outputs (subprocess pipes, misdecoded files), and the browser renders them leniently, so nobody notices. nbconvert serializes strictly and the first non-UTF-8 byte raises. The bug is always in the data that produced the output, not in nbconvert's encoder.

Edge cases / pitfalls

• Clearing outputs fixes the export but the next run recreates the bad bytes. Fix the cell's decoding for a durable fix.
• Subprocess output decoded with the wrong codec is the classic source. Set the locale or decode explicitly in the cell.
• errors='replace' in the cell hides the bad bytes with replacement characters. Fine for logs, bad for data.
• Binary outputs (images) are base64, not raw bytes, and are fine. The problem is always text outputs with bad bytes.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Hpb51a5W817A1VdxDB519Q