Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error

John6666
Wed, 06 Nov 2024 08:10:09 +0000

Many of the HF libraries use hard-coded file names, so sometimes they work and sometimes they don’t. If it works locally but not online, the problem is often with the file names, their placement, or the YAML part of README.md (the de facto repository configuration file).
Also, in your case, since you’re using a gated model, there’s a chance that the error is occurring because you’re failing to pass the token.
Even when calling the tokenizer, you need a token to read the repo.

If it seems like a bug in the library, you’ll need to find a way to make the error more visible and identify the bug itself. Or you could find the part that’s causing the problem and bypass it.
In this case, I’ll go read the github…