Feature Extraction
sentence-transformers
Safetensors
Transformers
English
new
fill-mask
learned sparse
opensearch
retrieval
passage-retrieval
document-expansion
bag-of-words
sparse-encoder
sparse
asymmetric
inference-free
custom_code
text-embeddings-inference
Instructions to use opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte with sentence-transformers:
from sentence_transformers import SparseEncoder model = SparseEncoder("opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte", trust_remote_code=True) queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Transformers
How to use opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Zhichao Geng commited on
Commit ·
9bdfa3d
1
Parent(s): 57d8a8f
Add file & weights
Browse files- README.md +163 -3
- config.json +40 -0
- idf.json +0 -0
- model.safetensors +3 -0
- query_token_weights.txt +0 -0
- special_tokens_map.json +7 -0
- tokenizer.json +0 -0
- tokenizer_config.json +55 -0
- vocab.txt +0 -0
README.md
CHANGED
|
@@ -1,3 +1,163 @@
|
|
| 1 |
-
---
|
| 2 |
-
|
| 3 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language: en
|
| 3 |
+
license: apache-2.0
|
| 4 |
+
tags:
|
| 5 |
+
- learned sparse
|
| 6 |
+
- opensearch
|
| 7 |
+
- transformers
|
| 8 |
+
- retrieval
|
| 9 |
+
- passage-retrieval
|
| 10 |
+
- document-expansion
|
| 11 |
+
- bag-of-words
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# opensearch-neural-sparse-encoding-doc-v3-distill
|
| 15 |
+
|
| 16 |
+
## Select the model
|
| 17 |
+
The model should be selected considering search relevance, model inference and retrieval efficiency(FLOPS). We benchmark models' performance on a subset of BEIR benchmark: TrecCovid,NFCorpus,NQ,HotpotQA,FiQA,ArguAna,Touche,DBPedia,SCIDOCS,FEVER,Climate FEVER,SciFact,Quora.
|
| 18 |
+
|
| 19 |
+
Overall, the v3 series of models have better search relevance, efficiency and inference speed than the v1 and v2 series. The specific advantages and disadvantages may vary across different datasets.
|
| 20 |
+
|
| 21 |
+
| Model | Inference-free for Retrieval | Model Parameters | AVG NDCG@10 | AVG FLOPS |
|
| 22 |
+
|-------|------------------------------|------------------|-------------|-----------|
|
| 23 |
+
| [opensearch-neural-sparse-encoding-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v1) | | 133M | 0.524 | 11.4 |
|
| 24 |
+
| [opensearch-neural-sparse-encoding-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v2-distill) | | 67M | 0.528 | 8.3 |
|
| 25 |
+
| [opensearch-neural-sparse-encoding-doc-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v1) | ✔️ | 133M | 0.490 | 2.3 |
|
| 26 |
+
| [opensearch-neural-sparse-encoding-doc-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill) | ✔️ | 67M | 0.504 | 1.8 |
|
| 27 |
+
| [opensearch-neural-sparse-encoding-doc-v2-mini](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v2-mini) | ✔️ | 23M | 0.497 | 1.7 |
|
| 28 |
+
| [opensearch-neural-sparse-encoding-doc-v3-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill) | ✔️ | 67M | 0.517 | 1.8 |
|
| 29 |
+
|
| 30 |
+
## Overview
|
| 31 |
+
- **Paper**: [Exploring $\ell_0$ Sparsification for Inference-free Sparse Retrievers ](https://arxiv.org/abs/2504.14839)
|
| 32 |
+
- **Codes**: [opensearch-sparse-model-tuning-sample](https://github.com/zhichao-aws/opensearch-sparse-model-tuning-sample/tree/l0_enhance)
|
| 33 |
+
|
| 34 |
+
This is a learned sparse retrieval model. It encodes the documents to 30522 dimensional **sparse vectors**. For queries, it just use a tokenizer and a weight look-up table to generate sparse vectors. The non-zero dimension index means the corresponding token in the vocabulary, and the weight means the importance of the token. And the similarity score is the inner product of query/document sparse vectors.
|
| 35 |
+
|
| 36 |
+
The training datasets includes MS MARCO, eli5_question_answer, squad_pairs, WikiAnswers, yahoo_answers_title_question, gooaq_pairs, stackexchange_duplicate_questions_body_body, wikihow, S2ORC_title_abstract, stackexchange_duplicate_questions_title-body_title-body, yahoo_answers_question_answer, searchQA_top5_snippets, stackexchange_duplicate_questions_title_title, yahoo_answers_title_answer, fever, fiqa, hotpotqa, nfcorpus, scifact.
|
| 37 |
+
|
| 38 |
+
OpenSearch neural sparse feature supports learned sparse retrieval with lucene inverted index. Link: https://opensearch.org/docs/latest/query-dsl/specialized/neural-sparse/. The indexing and search can be performed with OpenSearch high-level API.
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
## Usage (HuggingFace)
|
| 42 |
+
This model is supposed to run inside OpenSearch cluster. But you can also use it outside the cluster, with HuggingFace models API.
|
| 43 |
+
|
| 44 |
+
```python
|
| 45 |
+
import json
|
| 46 |
+
import itertools
|
| 47 |
+
import torch
|
| 48 |
+
|
| 49 |
+
from transformers import AutoModelForMaskedLM, AutoTokenizer
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
# get sparse vector from dense vectors with shape batch_size * seq_len * vocab_size
|
| 53 |
+
def get_sparse_vector(feature, output):
|
| 54 |
+
values, _ = torch.max(output*feature["attention_mask"].unsqueeze(-1), dim=1)
|
| 55 |
+
# note we update the activation for v3 model
|
| 56 |
+
values = torch.log(1 + torch.log(1 + torch.relu(values)))
|
| 57 |
+
values[:,special_token_ids] = 0
|
| 58 |
+
return values
|
| 59 |
+
|
| 60 |
+
# transform the sparse vector to a dict of (token, weight)
|
| 61 |
+
def transform_sparse_vector_to_dict(sparse_vector):
|
| 62 |
+
sample_indices,token_indices=torch.nonzero(sparse_vector,as_tuple=True)
|
| 63 |
+
non_zero_values = sparse_vector[(sample_indices,token_indices)].tolist()
|
| 64 |
+
number_of_tokens_for_each_sample = torch.bincount(sample_indices).cpu().tolist()
|
| 65 |
+
tokens = [transform_sparse_vector_to_dict.id_to_token[_id] for _id in token_indices.tolist()]
|
| 66 |
+
|
| 67 |
+
output = []
|
| 68 |
+
end_idxs = list(itertools.accumulate([0]+number_of_tokens_for_each_sample))
|
| 69 |
+
for i in range(len(end_idxs)-1):
|
| 70 |
+
token_strings = tokens[end_idxs[i]:end_idxs[i+1]]
|
| 71 |
+
weights = non_zero_values[end_idxs[i]:end_idxs[i+1]]
|
| 72 |
+
output.append(dict(zip(token_strings, weights)))
|
| 73 |
+
return output
|
| 74 |
+
|
| 75 |
+
# download the idf file from model hub. idf is used to give weights for query tokens
|
| 76 |
+
def get_tokenizer_idf(tokenizer):
|
| 77 |
+
from huggingface_hub import hf_hub_download
|
| 78 |
+
local_cached_path = hf_hub_download(repo_id="opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill", filename="idf.json")
|
| 79 |
+
with open(local_cached_path) as f:
|
| 80 |
+
idf = json.load(f)
|
| 81 |
+
idf_vector = [0]*tokenizer.vocab_size
|
| 82 |
+
for token,weight in idf.items():
|
| 83 |
+
_id = tokenizer._convert_token_to_id_with_added_voc(token)
|
| 84 |
+
idf_vector[_id]=weight
|
| 85 |
+
return torch.tensor(idf_vector)
|
| 86 |
+
|
| 87 |
+
# load the model
|
| 88 |
+
model = AutoModelForMaskedLM.from_pretrained("opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill")
|
| 89 |
+
tokenizer = AutoTokenizer.from_pretrained("opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill")
|
| 90 |
+
idf = get_tokenizer_idf(tokenizer)
|
| 91 |
+
|
| 92 |
+
# set the special tokens and id_to_token transform for post-process
|
| 93 |
+
special_token_ids = [tokenizer.vocab[token] for token in tokenizer.special_tokens_map.values()]
|
| 94 |
+
get_sparse_vector.special_token_ids = special_token_ids
|
| 95 |
+
id_to_token = ["" for i in range(tokenizer.vocab_size)]
|
| 96 |
+
for token, _id in tokenizer.vocab.items():
|
| 97 |
+
id_to_token[_id] = token
|
| 98 |
+
transform_sparse_vector_to_dict.id_to_token = id_to_token
|
| 99 |
+
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
query = "What's the weather in ny now?"
|
| 103 |
+
document = "Currently New York is rainy."
|
| 104 |
+
|
| 105 |
+
# encode the query
|
| 106 |
+
feature_query = tokenizer([query], padding=True, truncation=True, return_tensors='pt', return_token_type_ids=False)
|
| 107 |
+
input_ids = feature_query["input_ids"]
|
| 108 |
+
batch_size = input_ids.shape[0]
|
| 109 |
+
query_vector = torch.zeros(batch_size, tokenizer.vocab_size)
|
| 110 |
+
query_vector[torch.arange(batch_size).unsqueeze(-1), input_ids] = 1
|
| 111 |
+
query_sparse_vector = query_vector*idf
|
| 112 |
+
|
| 113 |
+
# encode the document
|
| 114 |
+
feature_document = tokenizer([document], padding=True, truncation=True, return_tensors='pt', return_token_type_ids=False)
|
| 115 |
+
output = model(**feature_document)[0]
|
| 116 |
+
document_sparse_vector = get_sparse_vector(feature_document, output)
|
| 117 |
+
|
| 118 |
+
|
| 119 |
+
# get similarity score
|
| 120 |
+
sim_score = torch.matmul(query_sparse_vector[0],document_sparse_vector[0])
|
| 121 |
+
print(sim_score) # tensor(11.1105, grad_fn=<DotBackward0>)
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
query_token_weight = transform_sparse_vector_to_dict(query_sparse_vector)[0]
|
| 125 |
+
document_query_token_weight = transform_sparse_vector_to_dict(document_sparse_vector)[0]
|
| 126 |
+
for token in sorted(query_token_weight, key=lambda x:query_token_weight[x], reverse=True):
|
| 127 |
+
if token in document_query_token_weight:
|
| 128 |
+
print("score in query: %.4f, score in document: %.4f, token: %s"%(query_token_weight[token],document_query_token_weight[token],token))
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
|
| 132 |
+
# result:
|
| 133 |
+
# score in query: 5.7729, score in document: 0.8049, token: ny
|
| 134 |
+
# score in query: 4.5684, score in document: 0.9710, token: weather
|
| 135 |
+
# score in query: 3.5895, score in document: 0.4720, token: now
|
| 136 |
+
# score in query: 3.3313, score in document: 0.0286, token: ?
|
| 137 |
+
# score in query: 2.7699, score in document: 0.0787, token: what
|
| 138 |
+
# score in query: 0.4989, score in document: 0.0417, token: in
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
The above code sample shows an example of neural sparse search. Although there is no overlap token in original query and document, but this model performs a good match.
|
| 142 |
+
|
| 143 |
+
## Detailed Search Relevance
|
| 144 |
+
|
| 145 |
+
<div style="overflow-x: auto;">
|
| 146 |
+
|
| 147 |
+
| Model | Average | Trec Covid | NFCorpus | NQ | HotpotQA | FiQA | ArguAna | Touche | DBPedia | SCIDOCS | FEVER | Climate FEVER | SciFact | Quora |
|
| 148 |
+
|-------|---------|------------|----------|----|----------|------|---------|--------|---------|---------|-------|---------------|---------|-------|
|
| 149 |
+
| [opensearch-neural-sparse-encoding-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v1) | 0.524 | 0.771 | 0.360 | 0.553 | 0.697 | 0.376 | 0.508 | 0.278 | 0.447 | 0.164 | 0.821 | 0.263 | 0.723 | 0.856 |
|
| 150 |
+
| [opensearch-neural-sparse-encoding-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v2-distill) | 0.528 | 0.775 | 0.347 | 0.561 | 0.685 | 0.374 | 0.551 | 0.278 | 0.435 | 0.173 | 0.849 | 0.249 | 0.722 | 0.863 |
|
| 151 |
+
| [opensearch-neural-sparse-encoding-doc-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v1) | 0.490 | 0.707 | 0.352 | 0.521 | 0.677 | 0.344 | 0.461 | 0.294 | 0.412 | 0.154 | 0.743 | 0.202 | 0.716 | 0.788 |
|
| 152 |
+
| [opensearch-neural-sparse-encoding-doc-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v2-distill) | 0.504 | 0.690 | 0.343 | 0.528 | 0.675 | 0.357 | 0.496 | 0.287 | 0.418 | 0.166 | 0.818 | 0.224 | 0.715 | 0.841 |
|
| 153 |
+
| [opensearch-neural-sparse-encoding-doc-v2-mini](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v2-mini) | 0.497 | 0.709 | 0.336 | 0.510 | 0.666 | 0.338 | 0.480 | 0.285 | 0.407 | 0.164 | 0.812 | 0.216 | 0.699 | 0.837 |
|
| 154 |
+
| [opensearch-neural-sparse-encoding-doc-v3-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill) | 0.517 | 0.724 | 0.345 | 0.544 | 0.694 | 0.356 | 0.520 | 0.294 | 0.424 | 0.163 | 0.845 | 0.239 | 0.708 | 0.863 |
|
| 155 |
+
</div>
|
| 156 |
+
|
| 157 |
+
## License
|
| 158 |
+
|
| 159 |
+
This project is licensed under the [Apache v2.0 License](https://github.com/opensearch-project/neural-search/blob/main/LICENSE).
|
| 160 |
+
|
| 161 |
+
## Copyright
|
| 162 |
+
|
| 163 |
+
Copyright OpenSearch Contributors. See [NOTICE](https://github.com/opensearch-project/neural-search/blob/main/NOTICE) for details.
|
config.json
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"architectures": [
|
| 3 |
+
"NewForMaskedLM"
|
| 4 |
+
],
|
| 5 |
+
"attention_probs_dropout_prob": 0.0,
|
| 6 |
+
"auto_map": {
|
| 7 |
+
"AutoConfig": "Alibaba-NLP/new-impl--configuration.NewConfig",
|
| 8 |
+
"AutoModelForMaskedLM": "Alibaba-NLP/new-impl--modeling.NewForMaskedLM",
|
| 9 |
+
"AutoModel": "Alibaba-NLP/new-impl--modeling.NewModel",
|
| 10 |
+
"AutoModelForMultipleChoice": "Alibaba-NLP/new-impl--modeling.NewForMultipleChoice",
|
| 11 |
+
"AutoModelForQuestionAnswering": "Alibaba-NLP/new-impl--modeling.NewForQuestionAnswering",
|
| 12 |
+
"AutoModelForSequenceClassification": "Alibaba-NLP/new-impl--modeling.NewForSequenceClassification",
|
| 13 |
+
"AutoModelForTokenClassification": "Alibaba-NLP/new-impl--modeling.NewForTokenClassification"
|
| 14 |
+
},
|
| 15 |
+
"classifier_dropout": 0.1,
|
| 16 |
+
"hidden_act": "gelu",
|
| 17 |
+
"hidden_dropout_prob": 0.1,
|
| 18 |
+
"hidden_size": 768,
|
| 19 |
+
"initializer_range": 0.02,
|
| 20 |
+
"intermediate_size": 3072,
|
| 21 |
+
"layer_norm_eps": 1e-12,
|
| 22 |
+
"layer_norm_type": "layer_norm",
|
| 23 |
+
"logn_attention_clip1": false,
|
| 24 |
+
"logn_attention_scale": false,
|
| 25 |
+
"max_position_embeddings": 8192,
|
| 26 |
+
"model_type": "new",
|
| 27 |
+
"num_attention_heads": 12,
|
| 28 |
+
"num_hidden_layers": 12,
|
| 29 |
+
"pack_qkv": true,
|
| 30 |
+
"pad_token_id": 0,
|
| 31 |
+
"position_embedding_type": "rope",
|
| 32 |
+
"rope_scaling": null,
|
| 33 |
+
"rope_theta": 500000,
|
| 34 |
+
"torch_dtype": "float32",
|
| 35 |
+
"transformers_version": "4.51.3",
|
| 36 |
+
"type_vocab_size": 0,
|
| 37 |
+
"unpad_inputs": false,
|
| 38 |
+
"use_memory_efficient_attention": false,
|
| 39 |
+
"vocab_size": 30522
|
| 40 |
+
}
|
idf.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e667add348c81c989f295b7c1d6ca9e910f6c12641a5d1ebe688b66af19cfb1f
|
| 3 |
+
size 549592272
|
query_token_weights.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
special_tokens_map.json
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"cls_token": "[CLS]",
|
| 3 |
+
"mask_token": "[MASK]",
|
| 4 |
+
"pad_token": "[PAD]",
|
| 5 |
+
"sep_token": "[SEP]",
|
| 6 |
+
"unk_token": "[UNK]"
|
| 7 |
+
}
|
tokenizer.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
tokenizer_config.json
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"added_tokens_decoder": {
|
| 3 |
+
"0": {
|
| 4 |
+
"content": "[PAD]",
|
| 5 |
+
"lstrip": false,
|
| 6 |
+
"normalized": false,
|
| 7 |
+
"rstrip": false,
|
| 8 |
+
"single_word": false,
|
| 9 |
+
"special": true
|
| 10 |
+
},
|
| 11 |
+
"100": {
|
| 12 |
+
"content": "[UNK]",
|
| 13 |
+
"lstrip": false,
|
| 14 |
+
"normalized": false,
|
| 15 |
+
"rstrip": false,
|
| 16 |
+
"single_word": false,
|
| 17 |
+
"special": true
|
| 18 |
+
},
|
| 19 |
+
"101": {
|
| 20 |
+
"content": "[CLS]",
|
| 21 |
+
"lstrip": false,
|
| 22 |
+
"normalized": false,
|
| 23 |
+
"rstrip": false,
|
| 24 |
+
"single_word": false,
|
| 25 |
+
"special": true
|
| 26 |
+
},
|
| 27 |
+
"102": {
|
| 28 |
+
"content": "[SEP]",
|
| 29 |
+
"lstrip": false,
|
| 30 |
+
"normalized": false,
|
| 31 |
+
"rstrip": false,
|
| 32 |
+
"single_word": false,
|
| 33 |
+
"special": true
|
| 34 |
+
},
|
| 35 |
+
"103": {
|
| 36 |
+
"content": "[MASK]",
|
| 37 |
+
"lstrip": false,
|
| 38 |
+
"normalized": false,
|
| 39 |
+
"rstrip": false,
|
| 40 |
+
"single_word": false,
|
| 41 |
+
"special": true
|
| 42 |
+
}
|
| 43 |
+
},
|
| 44 |
+
"clean_up_tokenization_spaces": true,
|
| 45 |
+
"cls_token": "[CLS]",
|
| 46 |
+
"do_lower_case": true,
|
| 47 |
+
"mask_token": "[MASK]",
|
| 48 |
+
"model_max_length": 512,
|
| 49 |
+
"pad_token": "[PAD]",
|
| 50 |
+
"sep_token": "[SEP]",
|
| 51 |
+
"strip_accents": null,
|
| 52 |
+
"tokenize_chinese_chars": true,
|
| 53 |
+
"tokenizer_class": "DistilBertTokenizer",
|
| 54 |
+
"unk_token": "[UNK]"
|
| 55 |
+
}
|
vocab.txt
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|