Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error

antoshka1608
Wed, 06 Nov 2024 07:21:11 +0000

Sorry, for, probably, misleading you, its not transformer’s tokenizer, but my custom ( i did it, inherited from PreTrainedTokenizer)
And yes, i reinstalled transformers mant times, doesnt work.

After your reply, i thought that this is might be issue with name duplication, but no. With new name of my tokenizer the same error(but of course with new tokenizer name ) .