| --- |
| library_name: sentence-transformers |
| pipeline_tag: sentence-similarity |
| tags: |
| - sentence-transformers |
| - feature-extraction |
| - sentence-similarity |
| - transformers |
| - phobert |
| - vietnamese |
| - sentence-embedding |
| - onnx |
| license: apache-2.0 |
| language: |
| - vi |
| metrics: |
| - pearsonr |
| - spearmanr |
| --- |
| ## Model Description: |
| [**vietnamese-embedding**](https://huggingface.co/dangvantuan/vietnamese-embedding) is the Embedding Model for Vietnamese language. This model is a specialized sentence-embedding trained specifically for the Vietnamese language, leveraging the robust capabilities of PhoBERT, a pre-trained language model based on the RoBERTa architecture. |
| The model utilizes PhoBERT to encode Vietnamese sentences into a 768-dimensional vector space, facilitating a wide range of applications from semantic search to text clustering. The embeddings capture the nuanced meanings of Vietnamese sentences, reflecting both the lexical and contextual layers of the language. |
|
|
| ## Full Model Architecture |
| ``` |
| SentenceTransformer( |
| (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: RobertaModel |
| (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
| ) |
| ``` |
| ## Training and Fine-tuning process |
| The model underwent a rigorous four-stage training and fine-tuning process, each tailored to enhance its ability to generate precise and contextually relevant sentence embeddings for the Vietnamese language. Below is an outline of these stages: |
| #### Stage 1: Initial Training |
| - Dataset: [ViNLI-SimCSE-supervised](https://huggingface.co/datasets/anti-ai/ViNLI-SimCSE-supervised) |
| - Method: Trained using the [SimCSE approach](https://arxiv.org/abs/2104.08821) which employs a supervised contrastive learning framework. The model was optimized using [Triplet Loss](https://www.sbert.net/docs/package_reference/losses.html#tripletloss) to effectively learn from high-quality annotated sentence pairs. |
| #### Stage 2: Continued Fine-tuning |
| - Dataset: [XNLI-vn ](https://huggingface.co/datasets/xnli/viewer/vi) |
| - Method: Continued fine-tuning using Multi-Negative Ranking Loss. This stage focused on improving the model's ability to discern and rank nuanced differences in sentence semantics. |
| ### Stage 3: Continued Fine-tuning for Semantic Textual Similarity on STS Benchmark |
| - Dataset: [STSB-vn](https://huggingface.co/datasets/doanhieung/vi-stsbenchmark) |
| - Method: Fine-tuning specifically for the semantic textual similarity benchmark using Siamese BERT-Networks configured with the 'sentence-transformers' library. This stage honed the model's precision in capturing semantic similarity across various types of Vietnamese texts. |
| ### Stage 4: Advanced Augmentation Fine-tuning |
| - Dataset: STSB-vn with generate [silver sample from gold sample](https://www.sbert.net/examples/training/data_augmentation/README.html) |
| - Method: Employed an advanced strategy using [Augmented SBERT](https://arxiv.org/abs/2010.08240) with Pair Sampling Strategies, integrating both Cross-Encoder and Bi-Encoder models. This stage further refined the embeddings by enriching the training data dynamically, enhancing the model's robustness and accuracy in understanding and processing complex Vietnamese language constructs. |
|
|
|
|
| ## Usage: |
|
|
| Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: |
|
|
| ``` |
| pip install -U sentence-transformers |
| pip install -q pyvi |
| ``` |
|
|
| Then you can use the model like this: |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| from pyvi.ViTokenizer import tokenize |
| |
| sentences = ["Hà Nội là thủ đô của Việt Nam", "Đà Nẵng là thành phố du lịch"] |
| tokenizer_sent = [tokenize(sent) for sent in sentences] |
| |
| model = SentenceTransformer('dangvantuan/vietnamese-embedding') |
| embeddings = model.encode(tokenizer_sent) |
| print(embeddings) |
| |
| ``` |
|
|
|
|
| ## Evaluation |
| The model can be evaluated as follows on the [Vienamese data of stsb](https://huggingface.co/datasets/doanhieung/vi-stsbenchmark). |
|
|
| ```python |
| from sentence_transformers import SentenceTransformer |
| from sentence_transformers import SentenceTransformer |
| from sentence_transformers.readers import InputExample |
| from datasets import load_dataset |
| from pyvi.ViTokenizer import tokenize |
| def convert_dataset(dataset): |
| dataset_samples=[] |
| for df in dataset: |
| score = float(df['score'])/5.0 # Normalize score to range 0 ... 1 |
| inp_example = InputExample(texts=[tokenize(df['sentence1']), |
| tokenize(df['sentence2'])], label=score) |
| dataset_samples.append(inp_example) |
| return dataset_samples |
| |
| # Loading the dataset for evaluation |
| vi_sts = load_dataset("doanhieung/vi-stsbenchmark")["train"] |
| df_dev = vi_sts.filter(lambda example: example['split'] == 'dev') |
| df_test = vi_sts.filter(lambda example: example['split'] == 'test') |
| |
| # Convert the dataset for evaluation |
| |
| # For Dev set: |
| dev_samples = convert_dataset(df_dev) |
| val_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(dev_samples, name='sts-dev') |
| val_evaluator(model, output_path="./") |
| |
| # For Test set: |
| test_samples = convert_dataset(df_test) |
| test_evaluator = EmbeddingSimilarityEvaluator.from_input_examples(test_samples, name='sts-test') |
| test_evaluator(model, output_path="./") |
| ``` |
|
|
|
|
| ### Test Result: |
| The performance is measured using Pearson and Spearman correlation: |
| - On dev |
| | Model | Pearson correlation | Spearman correlation | #params | |
| | ------------- | ------------- | ------------- |------------- | |
| | [dangvantuan/vietnamese-embedding](dangvantuan/vietnamese-embedding)| 88.33 |88.20 | 135M| |
| | [VoVanPhuc/sup-SimCSE-VietNamese-phobert-base](https://huggingface.co/VoVanPhuc/sup-SimCSE-VietNamese-phobert-base) | 84.65|84.59 | 135M | |
| | [keepitreal/vietnamese-sbert](https://huggingface.co/keepitreal/vietnamese-sbert) | 84.51 | 84.44|135M | |
| | [bkai-foundation-models/vietnamese-bi-encoder](https://huggingface.co/bkai-foundation-models/vietnamese-bi-encoder) | 78.05 | 77.94|135M | |
|
|
| ### Metric for all dataset of [Semantic Textual Similarity on STS Benchmark](https://huggingface.co/datasets/anti-ai/ViSTS) |
| You can run an evaluation on this [Colab](https://colab.research.google.com/drive/1JZLWKiknSUnA92UY2RIhvS65WtP6sgqW?hl=fr#scrollTo=IkTAwPqxDTOK) |
|
|
| **Pearson score** |
| | Model | [STSB] | [STS12]| [STS13] | [STS14] | [STS15] | [STS16] | [SICK] | Mean | |
| |-----------------------------------------------------------|---------|----------|----------|----------|----------|----------|---------|--------| |
| | [dangvantuan/vietnamese-embedding](dangvantuan/vietnamese-embedding) |**84.87** |**87.23**| **85.39**| **82.94**| **86.91**| **79.39**| **82.77**| **84.21**| |
| | [VoVanPhuc/sup-SimCSE-VietNamese-phobert-base](https://huggingface.co/VoVanPhuc/sup-SimCSE-VietNamese-phobert-base) |81.52| 85.02| 78.22| 75.94| 81.53| 75.39| 77.75| 79.33| |
| | [keepitreal/vietnamese-sbert](https://huggingface.co/keepitreal/vietnamese-sbert) |80.54| 78.58| 80.75| 76.98| 82.57| 73.21| 80.16| 78.97| |
| | [bkai-foundation-models/vietnamese-bi-encoder](https://huggingface.co/bkai-foundation-models/vietnamese-bi-encoder) |73.30| 67.84| 71.69| 69.80| 78.40| 74.29| 76.01| 73.04| |
|
|
|
|
| **Spearman score** |
| | Model | [STSB] | [STS12]| [STS13] | [STS14] | [STS15] | [STS16] | [SICK] | Mean | |
| |-----------------------------------------------------------|---------|----------|----------|----------|----------|----------|---------|--------| |
| | [dangvantuan/vietnamese-embedding](dangvantuan/vietnamese-embedding) |**84.84**| **79.04**| **85.30**| **81.38**| **87.06**| **79.95**| **79.58**| **82.45**| |
| | [VoVanPhuc/sup-SimCSE-VietNamese-phobert-base](https://huggingface.co/VoVanPhuc/sup-SimCSE-VietNamese-phobert-base) |81.43| 76.51| 79.19| 74.91| 81.72| 76.57| 76.45| 78.11| |
| | [keepitreal/vietnamese-sbert](https://huggingface.co/keepitreal/vietnamese-sbert) |80.16| 69.08| 80.99| 73.67| 82.81| 74.30| 73.40| 76.34| |
| | [bkai-foundation-models/vietnamese-bi-encoder](https://huggingface.co/bkai-foundation-models/vietnamese-bi-encoder) |72.16| 63.86| 71.82| 66.20| 78.62| 74.24| 70.87| 71.11| |
|
|
| ## Citation |
|
|
|
|
| @article{reimers2019sentence, |
| title={Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks}, |
| author={Nils Reimers, Iryna Gurevych}, |
| journal={https://arxiv.org/abs/1908.10084}, |
| year={2019} |
| } |
| |
|
|
| @article{martin2020camembert, |
| title={CamemBERT: a Tasty French Language Mode}, |
| author={Martin, Louis and Muller, Benjamin and Su{\'a}rez, Pedro Javier Ortiz and Dupont, Yoann and Romary, Laurent and de la Clergerie, {\'E}ric Villemonte and Seddah, Djam{\'e} and Sagot, Beno{\^\i}t}, |
| journal={Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}, |
| year={2020} |
| } |
| @article{thakur2020augmented, |
| title={Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks}, |
| author={Thakur, Nandan and Reimers, Nils and Daxenberger, Johannes and Gurevych, Iryna}, |
| journal={arXiv e-prints}, |
| pages={arXiv--2010}, |
| year={2020} |
