Tokenizers documentation
Encode Inputs
Encode Inputs
Not supported by the rc0 bindings yet. rc0 loads a
tokenizer.jsonand encodes and decodes with it — building a tokenizer from its components, editing one, saving one and training are not exposed. They are coming soon, along with the other bindings. SeeREQUIRED_FOR_V1.mdfor the full list.
These types represent all the different kinds of input that a Tokenizer accepts
when using ~tokenizers.Tokenizer.encode_batch.
TextEncodeInput[[[ tokenizers.TextEncodeInput ]]]
tokenizers.TextEncodeInput Represents a textual input for encoding. Can be either:
- A single sequence: TextInputSequence
- A pair of sequences:
- A Tuple of TextInputSequence
- Or a List of TextInputSequence of size 2
alias of Union[str, Tuple[str, str], List[str]].
PreTokenizedEncodeInput[[[ tokenizers.PreTokenizedEncodeInput ]]]
tokenizers.PreTokenizedEncodeInput Represents a pre-tokenized input for encoding. Can be either:
- A single sequence: PreTokenizedInputSequence
- A pair of sequences:
- A Tuple of PreTokenizedInputSequence
- Or a List of PreTokenizedInputSequence of size 2
alias of Union[List[str], Tuple[str], Tuple[Union[List[str], Tuple[str]], Union[List[str], Tuple[str]]], List[Union[List[str], Tuple[str]]]].
EncodeInput[[[ tokenizers.EncodeInput ]]]
tokenizers.EncodeInput Represents all the possible types of input for encoding. Can be:
- When
is_pretokenized=False: TextEncodeInput - When
is_pretokenized=True: PreTokenizedEncodeInput
alias of Union[str, Tuple[str, str], List[str], Tuple[str], Tuple[Union[List[str], Tuple[str]], Union[List[str], Tuple[str]]], List[Union[List[str], Tuple[str]]]].