|
--- |
|
pipeline_tag: sentence-similarity |
|
language: fr |
|
datasets: |
|
- stsb_multi_mt |
|
tags: |
|
- Text |
|
- Sentence Similarity |
|
- Sentence-Embedding |
|
- camembert-base |
|
license: apache-2.0 |
|
model-index: |
|
- name: CrossEncoder-camembert-large by Van Tuan DANG |
|
results: |
|
- task: |
|
name: Sentence-Embedding |
|
type: Text Similarity |
|
dataset: |
|
name: Text Similarity fr |
|
type: stsb_multi_mt |
|
args: fr |
|
metrics: |
|
- name: Test Pearson correlation coefficient |
|
type: Pearson_correlation_coefficient |
|
value: 90.34 |
|
--- |
|
|
|
## Model |
|
|
|
Cross-Encoder Model for sentence-similarity |
|
|
|
This model was is an improvement over the [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) offering greater robustness and better performance |
|
|
|
## Training Data |
|
This model was trained on the [STS benchmark dataset](https://huggingface.co/datasets/stsb_multi_mt/viewer/fr/train) and has been combined with [Augmented SBERT](https://aclanthology.org/2021.naacl-main.28.pdf). The model benefits from Pair Sampling Strategies using two models: [CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) and [dangvantuan/sentence-camembert-large](https://huggingface.co/dangvantuan/sentence-camembert-large). The model will predict a score between 0 and 1 how for the semantic similarity of two sentences. |
|
|
|
|
|
## Usage (Sentence-Transformers) |
|
|
|
Using this model becomes easy when you have [sentence-transformers](https://www.SBERT.net) installed: |
|
|
|
``` |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can use the model like this: |
|
|
|
```python |
|
from sentence_transformers import CrossEncoder |
|
model = CrossEncoder('Lajavaness/CrossEncoder-camembert-large', max_length=512) |
|
scores = model.predict([('Un avion est en train de décoller.', "Un homme joue d'une grande flûte."), ("Un homme étale du fromage râpé sur une pizza.", "Une personne jette un chat au plafond") ]) |
|
|
|
``` |
|
## Evaluation |
|
The model can be evaluated as follows on the French test data of stsb. |
|
```python |
|
from sentence_transformers.readers import InputExample |
|
from sentence_transformers.cross_encoder.evaluation import CECorrelationEvaluator |
|
from datasets import load_dataset |
|
def convert_dataset(dataset): |
|
dataset_samples=[] |
|
for df in dataset: |
|
score = float(df['similarity_score'])/5.0 # Normalize score to range 0 ... 1 |
|
inp_example = InputExample(texts=[df['sentence1'], |
|
df['sentence2']], label=score) |
|
dataset_samples.append(inp_example) |
|
return dataset_samples |
|
|
|
# Loading the dataset for evaluation |
|
df_dev = load_dataset("stsb_multi_mt", name="fr", split="dev") |
|
df_test = load_dataset("stsb_multi_mt", name="fr", split="test") |
|
|
|
# Convert the dataset for evaluation |
|
|
|
# For Dev set: |
|
dev_samples = convert_dataset(df_dev) |
|
val_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name='sts-dev') |
|
val_evaluator(model, output_path="./") |
|
|
|
# For Test set, the Pearson and Spearman correlation are evaluated on many different benchmark datasets: |
|
|
|
test_samples = convert_dataset(df_test) |
|
test_evaluator = CECorrelationEvaluator.from_input_examples(test_samples, name='sts-test') |
|
test_evaluator(models, output_path="./") |
|
``` |
|
**Test Result**: |
|
The performance is measured using Pearson and Spearman correlation: |
|
|
|
- On dev |
|
|
|
| Model | Pearson correlation | Spearman correlation | #params | |
|
| ------------- | ------------- | ------------- |------------- | |
|
| [Lajavaness/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 90.34 |90.15 | 336M | |
|
| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large)| 90.11 |90.01 | 336M | |
|
|
|
- On test: |
|
|
|
|
|
**Pearson score** |
|
|
|
| Model | STS-B | STS12-fr | STS13-fr | STS14-fr | STS15-fr | STS16-fr | SICK-fr | |
|
|---------------------------------------|--------|----------|----------|----------|----------|----------|---------| |
|
| [Lajavaness/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) | 88.63 | 90.76 | 88.24 | 90.22 | 92.23 | 82.31 | 84.61 | |
|
| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) | 88.16 | 90.12 | 88.36 | 89.86 | 92.04 | 82.01 | 84.23 | |
|
|
|
**Spearman score** |
|
|
|
|
|
| Model | STS-B | STS12-fr | STS13-fr | STS14-fr | STS15-fr | STS16-fr | SICK-fr | |
|
|---------------------------------------|--------|----------|----------|----------|----------|----------|---------| |
|
| [Lajavaness/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) | 88.03 | 84.87 | 87.88 | 89.10 | 92.16 | 82.50 | 80.78 | |
|
| [dangvantuan/CrossEncoder-camembert-large](https://huggingface.co/dangvantuan/CrossEncoder-camembert-large) | 87.57 | 84.24 | 88.01 | 88.62 | 91.99 | 82.16 | 80.38 | |
|
|
