Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error

John6666
Wed, 06 Nov 2024 04:14:47 +0000

That error usually occurs when the transoformers library is out of date, but it’s hard to imagine that you’re using a version so old that BaseTokeniker isn’t defined.

pip install -U transformers

There could be some other cause, such as the manual now being out of date.