Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error

antoshka1608
Wed, 06 Nov 2024 08:03:35 +0000

Yeah, thank you for you respond, anyway!!
Actually, if i import my tokenizer directly from
directory of my project.

tokenizer = CustomBaseTokenizer.from_pretrained(hugging_face_name)

It works good, but probably it takes just vocab file and doesnt use another files, so we cant check anything here.

Ill try your idea about other’s works later.