How to use rahular/oeullm_tok with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("rahular/oeullm_tok", device_map="auto")
https://huggingface.co/rahular/oeullm_tok/resolve/main/README.md
hf download hf://rahular/oeullm_tok/README.md
curl -L -o README.md https://huggingface.co/rahular/oeullm_tok/resolve/main/README.md
library_name: transformers tags: []
OELLM SPM Tokenizer with 128K tokens. It is trained with a config similar to that of GemmaTokenizer.
GemmaTokenizer
