Issue with Loading Custom Tokenizer: Tokenizer class BaseTokenizer does not exist or is not currently imported Error

antoshka1608
Tue, 05 Nov 2024 19:46:18 +0000

I’m encountering an issue when trying to load my custom tokenizer from a model repository on the Hugging Face Hub. Despite following the documentation for custom tokenizers.

huggingface.co

antoshka1608/wordpiece-tokenizer-v1 at main

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

To load the tokenizer, I’m using:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(“antoshka1608/wordpiece-tokenizer-v1”, use_fast=False, trust_remote_code=True)

When loading the tokenizer, it downloads tokenizer_config.json and vocab.json but then fails with the Tokenizer class BaseTokenizer does not exist or is not currently imported error.

Has anyone else encountered this issue or have suggestions on what might be going wrong? Any guidance on troubleshooting this would be greatly appreciated!

Thank you for your help!