Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error

John6666
Wed, 06 Nov 2024 07:53:46 +0000

huggingface.co

CohereForAI/c4ai-command-r-plus · ValueError: Tokenizer class CohereTokenizer...

Running this example script gives an error

Oh, I see. The name is the same as the existing one, and the tokenizer is also a custom one. Have you encountered this bug?
Also, in the case of HF in general, there are cases where information that was true at the time is now false in the manual, so I think the easiest thing to do is to refer to the py or json of someone else’s model that is working. It’s tough if there is no similar model…
The next best thing is to refer to the code of the library itself.