LookupError: unknown encoding: 'b'utf8''

This is a regression in lxml 5.2.0/5.2.1: it hands parsel a bytes object where a string encoding name belongs, and Python's codec lookup chokes on the repr. Upgrade or downgrade lxml out of the broken range (python -m pip install -U lxml once a fixed release is out, or pin lxml==5.1.0) and the LookupError disappears.

The error

LookupError: unknown encoding: 'b'utf8''

Fix it

1. Check your lxml version

python -c "import lxml.etree; print(lxml.etree.LXML_VERSION)"

Expected: if it prints (5, 2, 0, 0) or (5, 2, 1, 0), you have the broken build.

2. Move lxml out of the broken range

python -m pip install -U lxml

Expected: installs 5.2.2 or newer, where the regression is fixed. If you must stay pinned, python -m pip install lxml==5.1.0 also avoids it.

3. Confirm the selector works

python -c "from parsel import Selector; print(Selector('hi').xpath('string()').get())"

Expected: prints hi with no LookupError.

When this applies

• the LookupError names the encoding 'b'utf8'' (note the doubled quotes, it is a bytes repr)
• it comes from parsel.Selector(...) or Scrapy's TextResponse
• it started right after an lxml upgrade to 5.2.x

When it does NOT apply

• LookupError: unknown encoding: 'utf8' (no b-prefix) is a genuinely misspelled encoding somewhere in your code
• import errors for lxml itself are the missing/broken lxml problem

Compatibility

Broken: lxml 5.2.0 and 5.2.1. Fixed: lxml 5.2.2+. parsel versions are all affected equally because the bug is underneath parsel in lxml.

Why it happens

lxml 5.2 changed how it reports the document encoding to callers, and in two releases it returned the raw bytes b'utf8' instead of the string 'utf8'. parsel passes that value to Python's codec machinery, which stringifies the bytes object into the nonsense name 'b'utf8'' and raises LookupError. Nothing is wrong with your HTML or your code.

Edge cases

• If your lockfile pins lxml==5.2.1, bump the pin. The regression is in lxml, not parsel, so upgrading parsel alone does nothing.
• Scrapy users hit this through TextResponse; the fix is the same lxml move, not a Scrapy upgrade.
• If you cannot touch lxml (vendored env), decoding the body yourself and passing text= instead of body= sidesteps the encoding detection path.

In gh:scrapy/parsel#307, Selector construction started raising LookupError: unknown encoding: 'b'utf8'' after the lxml 5.2.1 release, and the thread bisected it to the lxml regression. Moving lxml out of the 5.2.0/5.2.1 range resolved it for every reporter. Source: https://github.com/scrapy/parsel/issues/307