32000-vocab-v2 - a flexitok Collection

flexitok 's Collections

maddition_360_v2

fineweb2-hq-tokenizers-v3

multilingual_addition_tokenizers_minimal

multilingual_addition_tokenizers

script-based-tokenizers

fineweb2-hq-tokenizers-v2

fineweb2-hq-tokenizers-feat2

flexitok-mod-tokenizers-zero-padded

flexitok/mod-tokenizers

flexitok/fineweb2-hq-superset-tokenizers

Fineweb2-HQ-Tokenizers

merging-toksuite

32000-vocab-v2

updated Apr 15