ModuleNotFoundError: No module named 'tokenizers'

How to fix ModuleNotFoundError: No module named 'tokenizers'

Install the missing Rust tokenizer backend: python -m pip install tokenizers. transformers needs it for every *Fast tokenizer (BertTokenizerFast etc.). Make sure you install into the same environment that runs the code: python -m pip with the python you actually use.

The exact error

ModuleNotFoundError: No module named 'tokenizers'

Fix it

1. python -m pip install tokenizers

Expected: Successfully installed tokenizers

1. python -c "import tokenizers; print(tokenizers.__version__)"

Expected: prints a version

1. Retry your transformers import.

Expected: works

When this applies

transformers raises No module named 'tokenizers' or 'requires the Tokenizers library but it was not found'

When this does NOT apply

No module named 'transformers' (install transformers itself first)

Versions

any transformers with fast tokenizers

Why it happens

Fast tokenizers are implemented in Rust in the separate tokenizers package; minimal installs skip it.

Edge cases

conda users: conda install -c conda-forge tokenizers avoids Rust-toolchain build issues.