Hugging Face's logo

Muennighoff
/
SGPT-125M-weightedmean-nli-bitfit

Sentence Similarity
sentence-transformers
PyTorch
gpt_neo
feature-extraction
mteb
Eval Results
Inference Endpoints
Model card Files Community
2
SGPT-125M-weightedmean-nli-bitfit / README.md
Muennighoff's picture
Muennighoff
Update README.md
54862a5
preview code
|
raw
history blame
95.6 kB
---
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- feature-extraction
- sentence-similarity
- mteb
model-index:
- name: SGPT-125M-weightedmean-nli-bitfit
results:
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (en)
metrics:
- type: accuracy
value: 65.88059701492537
- type: ap
value: 28.685493163579785
- type: f1
value: 59.79951005816335
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (de)
metrics:
- type: accuracy
value: 59.07922912205568
- type: ap
value: 73.91887421019034
- type: f1
value: 56.6316368658711
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (en-ext)
metrics:
- type: accuracy
value: 64.91754122938531
- type: ap
value: 16.360681214864226
- type: f1
value: 53.126592061523766
- task:
type: Classification
dataset:
type: mteb/amazon_counterfactual
name: MTEB AmazonCounterfactualClassification (ja)
metrics:
- type: accuracy
value: 56.423982869378996
- type: ap
value: 12.143003571907899
- type: f1
value: 45.76363777987471
- task:
type: Classification
dataset:
type: mteb/amazon_polarity
name: MTEB AmazonPolarityClassification
metrics:
- type: accuracy
value: 74.938225
- type: ap
value: 69.58187110320567
- type: f1
value: 74.72744058439321
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (en)
metrics:
- type: accuracy
value: 35.098
- type: f1
value: 34.73265651435726
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (de)
metrics:
- type: accuracy
value: 24.516
- type: f1
value: 24.21748200448397
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (es)
metrics:
- type: accuracy
value: 29.097999999999995
- type: f1
value: 28.620040162757093
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (fr)
metrics:
- type: accuracy
value: 27.395999999999997
- type: f1
value: 27.146888644986284
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (ja)
metrics:
- type: accuracy
value: 21.724
- type: f1
value: 21.37230564276654
- task:
type: Classification
dataset:
type: mteb/amazon_reviews_multi
name: MTEB AmazonReviewsClassification (zh)
metrics:
- type: accuracy
value: 23.976
- type: f1
value: 23.741137981755482
- task:
type: Retrieval
dataset:
type: arguana
name: MTEB ArguAna
metrics:
- type: map_at_1
value: 13.442000000000002
- type: map_at_10
value: 24.275
- type: map_at_100
value: 25.588
- type: map_at_1000
value: 25.659
- type: map_at_3
value: 20.092
- type: map_at_5
value: 22.439999999999998
- type: ndcg_at_1
value: 13.442000000000002
- type: ndcg_at_10
value: 31.04
- type: ndcg_at_100
value: 37.529
- type: ndcg_at_1000
value: 39.348
- type: ndcg_at_3
value: 22.342000000000002
- type: ndcg_at_5
value: 26.595999999999997
- type: precision_at_1
value: 13.442000000000002
- type: precision_at_10
value: 5.299
- type: precision_at_100
value: 0.836
- type: precision_at_1000
value: 0.098
- type: precision_at_3
value: 9.625
- type: precision_at_5
value: 7.852
- type: recall_at_1
value: 13.442000000000002
- type: recall_at_10
value: 52.986999999999995
- type: recall_at_100
value: 83.64200000000001
- type: recall_at_1000
value: 97.795
- type: recall_at_3
value: 28.876
- type: recall_at_5
value: 39.26
- task:
type: Clustering
dataset:
type: mteb/arxiv-clustering-p2p
name: MTEB ArxivClusteringP2P
metrics:
- type: v_measure
value: 34.742482477870766
- task:
type: Clustering
dataset:
type: mteb/arxiv-clustering-s2s
name: MTEB ArxivClusteringS2S
metrics:
- type: v_measure
value: 24.67870651472156
- task:
type: Reranking
dataset:
type: mteb/askubuntudupquestions-reranking
name: MTEB AskUbuntuDupQuestions
metrics:
- type: map
value: 52.63439984994702
- type: mrr
value: 65.75704612408214
- task:
type: STS
dataset:
type: mteb/biosses-sts
name: MTEB BIOSSES
metrics:
- type: cos_sim_pearson
value: 72.78000135012542
- type: cos_sim_spearman
value: 70.92812216947605
- type: euclidean_pearson
value: 77.1169214949292
- type: euclidean_spearman
value: 77.10175681583313
- type: manhattan_pearson
value: 76.84527031837595
- type: manhattan_spearman
value: 77.0704308008438
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (de-en)
metrics:
- type: accuracy
value: 1.0960334029227559
- type: f1
value: 1.0925539318023658
- type: precision
value: 1.0908141962421711
- type: recall
value: 1.0960334029227559
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (fr-en)
metrics:
- type: accuracy
value: 0.02201188641866608
- type: f1
value: 0.02201188641866608
- type: precision
value: 0.02201188641866608
- type: recall
value: 0.02201188641866608
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (ru-en)
metrics:
- type: accuracy
value: 0.0
- type: f1
value: 0.0
- type: precision
value: 0.0
- type: recall
value: 0.0
- task:
type: BitextMining
dataset:
type: mteb/bucc-bitext-mining
name: MTEB BUCC (zh-en)
metrics:
- type: accuracy
value: 0.0
- type: f1
value: 0.0
- type: precision
value: 0.0
- type: recall
value: 0.0
- task:
type: Classification
dataset:
type: mteb/banking77
name: MTEB Banking77Classification
metrics:
- type: accuracy
value: 74.67857142857142
- type: f1
value: 74.61743413995573
- task:
type: Clustering
dataset:
type: mteb/biorxiv-clustering-p2p
name: MTEB BiorxivClusteringP2P
metrics:
- type: v_measure
value: 28.93427045246491
- task:
type: Clustering
dataset:
type: mteb/biorxiv-clustering-s2s
name: MTEB BiorxivClusteringS2S
metrics:
- type: v_measure
value: 23.080939123955474
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackAndroidRetrieval
metrics:
- type: map_at_1
value: 18.221999999999998
- type: map_at_10
value: 24.506
- type: map_at_100
value: 25.611
- type: map_at_1000
value: 25.758
- type: map_at_3
value: 22.264999999999997
- type: map_at_5
value: 23.698
- type: ndcg_at_1
value: 23.033
- type: ndcg_at_10
value: 28.719
- type: ndcg_at_100
value: 33.748
- type: ndcg_at_1000
value: 37.056
- type: ndcg_at_3
value: 25.240000000000002
- type: ndcg_at_5
value: 27.12
- type: precision_at_1
value: 23.033
- type: precision_at_10
value: 5.408
- type: precision_at_100
value: 1.004
- type: precision_at_1000
value: 0.158
- type: precision_at_3
value: 11.874
- type: precision_at_5
value: 8.927
- type: recall_at_1
value: 18.221999999999998
- type: recall_at_10
value: 36.355
- type: recall_at_100
value: 58.724
- type: recall_at_1000
value: 81.33500000000001
- type: recall_at_3
value: 26.334000000000003
- type: recall_at_5
value: 31.4
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackEnglishRetrieval
metrics:
- type: map_at_1
value: 12.058
- type: map_at_10
value: 16.051000000000002
- type: map_at_100
value: 16.772000000000002
- type: map_at_1000
value: 16.871
- type: map_at_3
value: 14.78
- type: map_at_5
value: 15.5
- type: ndcg_at_1
value: 15.35
- type: ndcg_at_10
value: 18.804000000000002
- type: ndcg_at_100
value: 22.346
- type: ndcg_at_1000
value: 25.007
- type: ndcg_at_3
value: 16.768
- type: ndcg_at_5
value: 17.692
- type: precision_at_1
value: 15.35
- type: precision_at_10
value: 3.51
- type: precision_at_100
value: 0.664
- type: precision_at_1000
value: 0.11100000000000002
- type: precision_at_3
value: 7.983
- type: precision_at_5
value: 5.656
- type: recall_at_1
value: 12.058
- type: recall_at_10
value: 23.644000000000002
- type: recall_at_100
value: 39.76
- type: recall_at_1000
value: 58.56
- type: recall_at_3
value: 17.541999999999998
- type: recall_at_5
value: 20.232
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackGamingRetrieval
metrics:
- type: map_at_1
value: 21.183
- type: map_at_10
value: 28.9
- type: map_at_100
value: 29.858
- type: map_at_1000
value: 29.953999999999997
- type: map_at_3
value: 26.58
- type: map_at_5
value: 27.912
- type: ndcg_at_1
value: 24.765
- type: ndcg_at_10
value: 33.339999999999996
- type: ndcg_at_100
value: 37.997
- type: ndcg_at_1000
value: 40.416000000000004
- type: ndcg_at_3
value: 29.044999999999998
- type: ndcg_at_5
value: 31.121
- type: precision_at_1
value: 24.765
- type: precision_at_10
value: 5.599
- type: precision_at_100
value: 0.8699999999999999
- type: precision_at_1000
value: 0.11499999999999999
- type: precision_at_3
value: 13.270999999999999
- type: precision_at_5
value: 9.367
- type: recall_at_1
value: 21.183
- type: recall_at_10
value: 43.875
- type: recall_at_100
value: 65.005
- type: recall_at_1000
value: 83.017
- type: recall_at_3
value: 32.232
- type: recall_at_5
value: 37.308
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackGisRetrieval
metrics:
- type: map_at_1
value: 11.350999999999999
- type: map_at_10
value: 14.953
- type: map_at_100
value: 15.623000000000001
- type: map_at_1000
value: 15.716
- type: map_at_3
value: 13.603000000000002
- type: map_at_5
value: 14.343
- type: ndcg_at_1
value: 12.429
- type: ndcg_at_10
value: 17.319000000000003
- type: ndcg_at_100
value: 20.990000000000002
- type: ndcg_at_1000
value: 23.899
- type: ndcg_at_3
value: 14.605
- type: ndcg_at_5
value: 15.89
- type: precision_at_1
value: 12.429
- type: precision_at_10
value: 2.701
- type: precision_at_100
value: 0.48700000000000004
- type: precision_at_1000
value: 0.078
- type: precision_at_3
value: 6.026
- type: precision_at_5
value: 4.3839999999999995
- type: recall_at_1
value: 11.350999999999999
- type: recall_at_10
value: 23.536
- type: recall_at_100
value: 40.942
- type: recall_at_1000
value: 64.05
- type: recall_at_3
value: 16.195
- type: recall_at_5
value: 19.264
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackMathematicaRetrieval
metrics:
- type: map_at_1
value: 8.08
- type: map_at_10
value: 11.691
- type: map_at_100
value: 12.312
- type: map_at_1000
value: 12.439
- type: map_at_3
value: 10.344000000000001
- type: map_at_5
value: 10.996
- type: ndcg_at_1
value: 10.697
- type: ndcg_at_10
value: 14.48
- type: ndcg_at_100
value: 18.160999999999998
- type: ndcg_at_1000
value: 21.886
- type: ndcg_at_3
value: 11.872
- type: ndcg_at_5
value: 12.834000000000001
- type: precision_at_1
value: 10.697
- type: precision_at_10
value: 2.811
- type: precision_at_100
value: 0.551
- type: precision_at_1000
value: 0.10200000000000001
- type: precision_at_3
value: 5.804
- type: precision_at_5
value: 4.154
- type: recall_at_1
value: 8.08
- type: recall_at_10
value: 20.235
- type: recall_at_100
value: 37.525999999999996
- type: recall_at_1000
value: 65.106
- type: recall_at_3
value: 12.803999999999998
- type: recall_at_5
value: 15.498999999999999
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackPhysicsRetrieval
metrics:
- type: map_at_1
value: 13.908999999999999
- type: map_at_10
value: 19.256
- type: map_at_100
value: 20.286
- type: map_at_1000
value: 20.429
- type: map_at_3
value: 17.399
- type: map_at_5
value: 18.398999999999997
- type: ndcg_at_1
value: 17.421
- type: ndcg_at_10
value: 23.105999999999998
- type: ndcg_at_100
value: 28.128999999999998
- type: ndcg_at_1000
value: 31.480999999999998
- type: ndcg_at_3
value: 19.789
- type: ndcg_at_5
value: 21.237000000000002
- type: precision_at_1
value: 17.421
- type: precision_at_10
value: 4.331
- type: precision_at_100
value: 0.839
- type: precision_at_1000
value: 0.131
- type: precision_at_3
value: 9.4
- type: precision_at_5
value: 6.776
- type: recall_at_1
value: 13.908999999999999
- type: recall_at_10
value: 31.086999999999996
- type: recall_at_100
value: 52.946000000000005
- type: recall_at_1000
value: 76.546
- type: recall_at_3
value: 21.351
- type: recall_at_5
value: 25.264999999999997
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackProgrammersRetrieval
metrics:
- type: map_at_1
value: 12.598
- type: map_at_10
value: 17.304
- type: map_at_100
value: 18.209
- type: map_at_1000
value: 18.328
- type: map_at_3
value: 15.784
- type: map_at_5
value: 16.669999999999998
- type: ndcg_at_1
value: 15.867999999999999
- type: ndcg_at_10
value: 20.623
- type: ndcg_at_100
value: 25.093
- type: ndcg_at_1000
value: 28.498
- type: ndcg_at_3
value: 17.912
- type: ndcg_at_5
value: 19.198
- type: precision_at_1
value: 15.867999999999999
- type: precision_at_10
value: 3.7670000000000003
- type: precision_at_100
value: 0.716
- type: precision_at_1000
value: 0.11800000000000001
- type: precision_at_3
value: 8.638
- type: precision_at_5
value: 6.21
- type: recall_at_1
value: 12.598
- type: recall_at_10
value: 27.144000000000002
- type: recall_at_100
value: 46.817
- type: recall_at_1000
value: 71.86099999999999
- type: recall_at_3
value: 19.231
- type: recall_at_5
value: 22.716
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackRetrieval
metrics:
- type: map_at_1
value: 12.738416666666666
- type: map_at_10
value: 17.235916666666668
- type: map_at_100
value: 18.063333333333333
- type: map_at_1000
value: 18.18433333333333
- type: map_at_3
value: 15.74775
- type: map_at_5
value: 16.57825
- type: ndcg_at_1
value: 15.487416666666665
- type: ndcg_at_10
value: 20.290166666666668
- type: ndcg_at_100
value: 24.41291666666666
- type: ndcg_at_1000
value: 27.586333333333336
- type: ndcg_at_3
value: 17.622083333333332
- type: ndcg_at_5
value: 18.859916666666667
- type: precision_at_1
value: 15.487416666666665
- type: precision_at_10
value: 3.6226666666666665
- type: precision_at_100
value: 0.6820833333333334
- type: precision_at_1000
value: 0.11216666666666666
- type: precision_at_3
value: 8.163749999999999
- type: precision_at_5
value: 5.865416666666667
- type: recall_at_1
value: 12.738416666666666
- type: recall_at_10
value: 26.599416666666663
- type: recall_at_100
value: 45.41258333333334
- type: recall_at_1000
value: 68.7565
- type: recall_at_3
value: 19.008166666666668
- type: recall_at_5
value: 22.24991666666667
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackStatsRetrieval
metrics:
- type: map_at_1
value: 12.307
- type: map_at_10
value: 15.440000000000001
- type: map_at_100
value: 16.033
- type: map_at_1000
value: 16.14
- type: map_at_3
value: 14.393
- type: map_at_5
value: 14.856
- type: ndcg_at_1
value: 14.571000000000002
- type: ndcg_at_10
value: 17.685000000000002
- type: ndcg_at_100
value: 20.882
- type: ndcg_at_1000
value: 23.888
- type: ndcg_at_3
value: 15.739
- type: ndcg_at_5
value: 16.391
- type: precision_at_1
value: 14.571000000000002
- type: precision_at_10
value: 2.883
- type: precision_at_100
value: 0.49100000000000005
- type: precision_at_1000
value: 0.08
- type: precision_at_3
value: 7.0040000000000004
- type: precision_at_5
value: 4.693
- type: recall_at_1
value: 12.307
- type: recall_at_10
value: 22.566
- type: recall_at_100
value: 37.469
- type: recall_at_1000
value: 60.550000000000004
- type: recall_at_3
value: 16.742
- type: recall_at_5
value: 18.634
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackTexRetrieval
metrics:
- type: map_at_1
value: 6.496
- type: map_at_10
value: 9.243
- type: map_at_100
value: 9.841
- type: map_at_1000
value: 9.946000000000002
- type: map_at_3
value: 8.395
- type: map_at_5
value: 8.872
- type: ndcg_at_1
value: 8.224
- type: ndcg_at_10
value: 11.24
- type: ndcg_at_100
value: 14.524999999999999
- type: ndcg_at_1000
value: 17.686
- type: ndcg_at_3
value: 9.617
- type: ndcg_at_5
value: 10.37
- type: precision_at_1
value: 8.224
- type: precision_at_10
value: 2.0820000000000003
- type: precision_at_100
value: 0.443
- type: precision_at_1000
value: 0.08499999999999999
- type: precision_at_3
value: 4.623
- type: precision_at_5
value: 3.331
- type: recall_at_1
value: 6.496
- type: recall_at_10
value: 15.310000000000002
- type: recall_at_100
value: 30.680000000000003
- type: recall_at_1000
value: 54.335
- type: recall_at_3
value: 10.691
- type: recall_at_5
value: 12.687999999999999
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackUnixRetrieval
metrics:
- type: map_at_1
value: 13.843
- type: map_at_10
value: 17.496000000000002
- type: map_at_100
value: 18.304000000000002
- type: map_at_1000
value: 18.426000000000002
- type: map_at_3
value: 16.225
- type: map_at_5
value: 16.830000000000002
- type: ndcg_at_1
value: 16.698
- type: ndcg_at_10
value: 20.301
- type: ndcg_at_100
value: 24.523
- type: ndcg_at_1000
value: 27.784
- type: ndcg_at_3
value: 17.822
- type: ndcg_at_5
value: 18.794
- type: precision_at_1
value: 16.698
- type: precision_at_10
value: 3.3579999999999997
- type: precision_at_100
value: 0.618
- type: precision_at_1000
value: 0.101
- type: precision_at_3
value: 7.898
- type: precision_at_5
value: 5.428999999999999
- type: recall_at_1
value: 13.843
- type: recall_at_10
value: 25.887999999999998
- type: recall_at_100
value: 45.028
- type: recall_at_1000
value: 68.991
- type: recall_at_3
value: 18.851000000000003
- type: recall_at_5
value: 21.462
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackWebmastersRetrieval
metrics:
- type: map_at_1
value: 13.757
- type: map_at_10
value: 19.27
- type: map_at_100
value: 20.461
- type: map_at_1000
value: 20.641000000000002
- type: map_at_3
value: 17.865000000000002
- type: map_at_5
value: 18.618000000000002
- type: ndcg_at_1
value: 16.996
- type: ndcg_at_10
value: 22.774
- type: ndcg_at_100
value: 27.675
- type: ndcg_at_1000
value: 31.145
- type: ndcg_at_3
value: 20.691000000000003
- type: ndcg_at_5
value: 21.741
- type: precision_at_1
value: 16.996
- type: precision_at_10
value: 4.545
- type: precision_at_100
value: 1.036
- type: precision_at_1000
value: 0.185
- type: precision_at_3
value: 10.145
- type: precision_at_5
value: 7.391
- type: recall_at_1
value: 13.757
- type: recall_at_10
value: 28.233999999999998
- type: recall_at_100
value: 51.05499999999999
- type: recall_at_1000
value: 75.35300000000001
- type: recall_at_3
value: 21.794
- type: recall_at_5
value: 24.614
- task:
type: Retrieval
dataset:
type: BeIR/cqadupstack
name: MTEB CQADupstackWordpressRetrieval
metrics:
- type: map_at_1
value: 9.057
- type: map_at_10
value: 12.720999999999998
- type: map_at_100
value: 13.450000000000001
- type: map_at_1000
value: 13.564000000000002
- type: map_at_3
value: 11.34
- type: map_at_5
value: 12.245000000000001
- type: ndcg_at_1
value: 9.797
- type: ndcg_at_10
value: 15.091
- type: ndcg_at_100
value: 18.886
- type: ndcg_at_1000
value: 22.29
- type: ndcg_at_3
value: 12.365
- type: ndcg_at_5
value: 13.931
- type: precision_at_1
value: 9.797
- type: precision_at_10
value: 2.477
- type: precision_at_100
value: 0.466
- type: precision_at_1000
value: 0.082
- type: precision_at_3
value: 5.299
- type: precision_at_5
value: 4.067
- type: recall_at_1
value: 9.057
- type: recall_at_10
value: 21.319
- type: recall_at_100
value: 38.999
- type: recall_at_1000
value: 65.374
- type: recall_at_3
value: 14.331
- type: recall_at_5
value: 17.916999999999998
- task:
type: Retrieval
dataset:
type: climate-fever
name: MTEB ClimateFEVER
metrics:
- type: map_at_1
value: 3.714
- type: map_at_10
value: 6.926
- type: map_at_100
value: 7.879
- type: map_at_1000
value: 8.032
- type: map_at_3
value: 5.504
- type: map_at_5
value: 6.357
- type: ndcg_at_1
value: 8.86
- type: ndcg_at_10
value: 11.007
- type: ndcg_at_100
value: 16.154
- type: ndcg_at_1000
value: 19.668
- type: ndcg_at_3
value: 8.103
- type: ndcg_at_5
value: 9.456000000000001
- type: precision_at_1
value: 8.86
- type: precision_at_10
value: 3.7199999999999998
- type: precision_at_100
value: 0.9169999999999999
- type: precision_at_1000
value: 0.156
- type: precision_at_3
value: 6.254
- type: precision_at_5
value: 5.380999999999999
- type: recall_at_1
value: 3.714
- type: recall_at_10
value: 14.382
- type: recall_at_100
value: 33.166000000000004
- type: recall_at_1000
value: 53.444
- type: recall_at_3
value: 7.523000000000001
- type: recall_at_5
value: 10.91
- task:
type: Retrieval
dataset:
type: dbpedia-entity
name: MTEB DBPedia
metrics:
- type: map_at_1
value: 1.764
- type: map_at_10
value: 3.8600000000000003
- type: map_at_100
value: 5.457
- type: map_at_1000
value: 5.938000000000001
- type: map_at_3
value: 2.667
- type: map_at_5
value: 3.2199999999999998
- type: ndcg_at_1
value: 14.000000000000002
- type: ndcg_at_10
value: 10.868
- type: ndcg_at_100
value: 12.866
- type: ndcg_at_1000
value: 17.43
- type: ndcg_at_3
value: 11.943
- type: ndcg_at_5
value: 11.66
- type: precision_at_1
value: 19.25
- type: precision_at_10
value: 10.274999999999999
- type: precision_at_100
value: 3.527
- type: precision_at_1000
value: 0.9119999999999999
- type: precision_at_3
value: 14.917
- type: precision_at_5
value: 13.5
- type: recall_at_1
value: 1.764
- type: recall_at_10
value: 6.609
- type: recall_at_100
value: 17.616
- type: recall_at_1000
value: 33.085
- type: recall_at_3
value: 3.115
- type: recall_at_5
value: 4.605
- task:
type: Classification
dataset:
type: mteb/emotion
name: MTEB EmotionClassification
metrics:
- type: accuracy
value: 42.225
- type: f1
value: 37.563516542112104
- task:
type: Retrieval
dataset:
type: fever
name: MTEB FEVER
metrics:
- type: map_at_1
value: 11.497
- type: map_at_10
value: 15.744
- type: map_at_100
value: 16.3
- type: map_at_1000
value: 16.365
- type: map_at_3
value: 14.44
- type: map_at_5
value: 15.18
- type: ndcg_at_1
value: 12.346
- type: ndcg_at_10
value: 18.398999999999997
- type: ndcg_at_100
value: 21.399
- type: ndcg_at_1000
value: 23.442
- type: ndcg_at_3
value: 15.695
- type: ndcg_at_5
value: 17.027
- type: precision_at_1
value: 12.346
- type: precision_at_10
value: 2.798
- type: precision_at_100
value: 0.445
- type: precision_at_1000
value: 0.063
- type: precision_at_3
value: 6.586
- type: precision_at_5
value: 4.665
- type: recall_at_1
value: 11.497
- type: recall_at_10
value: 25.636
- type: recall_at_100
value: 39.894
- type: recall_at_1000
value: 56.181000000000004
- type: recall_at_3
value: 18.273
- type: recall_at_5
value: 21.474
- task:
type: Retrieval
dataset:
type: fiqa
name: MTEB FiQA2018
metrics:
- type: map_at_1
value: 3.637
- type: map_at_10
value: 6.084
- type: map_at_100
value: 6.9190000000000005
- type: map_at_1000
value: 7.1080000000000005
- type: map_at_3
value: 5.071
- type: map_at_5
value: 5.5649999999999995
- type: ndcg_at_1
value: 7.407
- type: ndcg_at_10
value: 8.94
- type: ndcg_at_100
value: 13.594999999999999
- type: ndcg_at_1000
value: 18.29
- type: ndcg_at_3
value: 7.393
- type: ndcg_at_5
value: 7.854
- type: precision_at_1
value: 7.407
- type: precision_at_10
value: 2.778
- type: precision_at_100
value: 0.75
- type: precision_at_1000
value: 0.154
- type: precision_at_3
value: 5.144
- type: precision_at_5
value: 3.981
- type: recall_at_1
value: 3.637
- type: recall_at_10
value: 11.821
- type: recall_at_100
value: 30.18
- type: recall_at_1000
value: 60.207
- type: recall_at_3
value: 6.839
- type: recall_at_5
value: 8.649
- task:
type: Retrieval
dataset:
type: hotpotqa
name: MTEB HotpotQA
metrics:
- type: map_at_1
value: 9.676
- type: map_at_10
value: 13.350999999999999
- type: map_at_100
value: 13.919
- type: map_at_1000
value: 14.01
- type: map_at_3
value: 12.223
- type: map_at_5
value: 12.812000000000001
- type: ndcg_at_1
value: 19.352
- type: ndcg_at_10
value: 17.727
- type: ndcg_at_100
value: 20.837
- type: ndcg_at_1000
value: 23.412
- type: ndcg_at_3
value: 15.317
- type: ndcg_at_5
value: 16.436
- type: precision_at_1
value: 19.352
- type: precision_at_10
value: 3.993
- type: precision_at_100
value: 0.651
- type: precision_at_1000
value: 0.1
- type: precision_at_3
value: 9.669
- type: precision_at_5
value: 6.69
- type: recall_at_1
value: 9.676
- type: recall_at_10
value: 19.966
- type: recall_at_100
value: 32.573
- type: recall_at_1000
value: 49.905
- type: recall_at_3
value: 14.504
- type: recall_at_5
value: 16.725
- task:
type: Classification
dataset:
type: mteb/imdb
name: MTEB ImdbClassification
metrics:
- type: accuracy
value: 62.895999999999994
- type: ap
value: 58.47769349850157
- type: f1
value: 62.67885149592086
- task:
type: Retrieval
dataset:
type: msmarco
name: MTEB MSMARCO
metrics:
- type: map_at_1
value: 2.88
- type: map_at_10
value: 4.914000000000001
- type: map_at_100
value: 5.459
- type: map_at_1000
value: 5.538
- type: map_at_3
value: 4.087
- type: map_at_5
value: 4.518
- type: ndcg_at_1
value: 2.937
- type: ndcg_at_10
value: 6.273
- type: ndcg_at_100
value: 9.426
- type: ndcg_at_1000
value: 12.033000000000001
- type: ndcg_at_3
value: 4.513
- type: ndcg_at_5
value: 5.292
- type: precision_at_1
value: 2.937
- type: precision_at_10
value: 1.089
- type: precision_at_100
value: 0.27699999999999997
- type: precision_at_1000
value: 0.051000000000000004
- type: precision_at_3
value: 1.9290000000000003
- type: precision_at_5
value: 1.547
- type: recall_at_1
value: 2.88
- type: recall_at_10
value: 10.578
- type: recall_at_100
value: 26.267000000000003
- type: recall_at_1000
value: 47.589999999999996
- type: recall_at_3
value: 5.673
- type: recall_at_5
value: 7.545
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (en)
metrics:
- type: accuracy
value: 81.51846785225717
- type: f1
value: 81.648869152345
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (de)
metrics:
- type: accuracy
value: 60.37475345167653
- type: f1
value: 58.452649375517026
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (es)
metrics:
- type: accuracy
value: 67.36824549699799
- type: f1
value: 65.35927434998516
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (fr)
metrics:
- type: accuracy
value: 63.12871907297212
- type: f1
value: 61.37620329272278
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (hi)
metrics:
- type: accuracy
value: 47.04553603442094
- type: f1
value: 46.20389912644561
- task:
type: Classification
dataset:
type: mteb/mtop_domain
name: MTEB MTOPDomainClassification (th)
metrics:
- type: accuracy
value: 52.282097649186255
- type: f1
value: 50.75489206473579
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (en)
metrics:
- type: accuracy
value: 58.2421340629275
- type: f1
value: 40.11696046622642
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (de)
metrics:
- type: accuracy
value: 45.069033530571986
- type: f1
value: 30.468468273374967
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (es)
metrics:
- type: accuracy
value: 48.80920613742495
- type: f1
value: 32.65985375400447
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (fr)
metrics:
- type: accuracy
value: 44.337613529595984
- type: f1
value: 29.302047435606436
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (hi)
metrics:
- type: accuracy
value: 34.198637504481894
- type: f1
value: 22.063706032248408
- task:
type: Classification
dataset:
type: mteb/mtop_intent
name: MTEB MTOPIntentClassification (th)
metrics:
- type: accuracy
value: 43.11030741410488
- type: f1
value: 26.92408933648504
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (af)
metrics:
- type: accuracy
value: 37.79421654337593
- type: f1
value: 36.81580701507746
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (am)
metrics:
- type: accuracy
value: 23.722259583053127
- type: f1
value: 23.235269695764273
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ar)
metrics:
- type: accuracy
value: 29.64021519838601
- type: f1
value: 28.273175327650137
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (az)
metrics:
- type: accuracy
value: 39.4754539340955
- type: f1
value: 39.25997361415121
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (bn)
metrics:
- type: accuracy
value: 26.550100874243444
- type: f1
value: 25.607924873522975
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (cy)
metrics:
- type: accuracy
value: 38.78278412911904
- type: f1
value: 37.64180582626517
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (da)
metrics:
- type: accuracy
value: 43.557498318762605
- type: f1
value: 41.35305173800667
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (de)
metrics:
- type: accuracy
value: 40.39340954942838
- type: f1
value: 38.33393219528934
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (el)
metrics:
- type: accuracy
value: 37.28648285137861
- type: f1
value: 36.64005906680284
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (en)
metrics:
- type: accuracy
value: 58.080026899798256
- type: f1
value: 56.49243881660991
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (es)
metrics:
- type: accuracy
value: 41.176866173503704
- type: f1
value: 40.66779962225799
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (fa)
metrics:
- type: accuracy
value: 36.422326832548755
- type: f1
value: 34.6441738042885
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (fi)
metrics:
- type: accuracy
value: 38.75588433086752
- type: f1
value: 37.26725894668694
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (fr)
metrics:
- type: accuracy
value: 43.67182246133153
- type: f1
value: 42.351846624566605
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (he)
metrics:
- type: accuracy
value: 31.980497646267658
- type: f1
value: 30.557928872809008
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (hi)
metrics:
- type: accuracy
value: 28.039677202420982
- type: f1
value: 28.428418145508306
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (hu)
metrics:
- type: accuracy
value: 38.13718897108272
- type: f1
value: 37.057406988196874
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (hy)
metrics:
- type: accuracy
value: 26.05245460659045
- type: f1
value: 25.25483953344816
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (id)
metrics:
- type: accuracy
value: 41.156691324815064
- type: f1
value: 40.83715033247605
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (is)
metrics:
- type: accuracy
value: 38.62811028917284
- type: f1
value: 37.67691901246032
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (it)
metrics:
- type: accuracy
value: 44.0383322125084
- type: f1
value: 43.77259010877456
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ja)
metrics:
- type: accuracy
value: 46.20712844653666
- type: f1
value: 44.66632875940824
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (jv)
metrics:
- type: accuracy
value: 37.60591795561533
- type: f1
value: 36.581071742378015
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ka)
metrics:
- type: accuracy
value: 24.47209145931405
- type: f1
value: 24.238209697895606
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (km)
metrics:
- type: accuracy
value: 26.23739071956961
- type: f1
value: 25.378783150845052
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (kn)
metrics:
- type: accuracy
value: 17.831203765971754
- type: f1
value: 17.275078420466343
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ko)
metrics:
- type: accuracy
value: 37.266308002689975
- type: f1
value: 36.92473791708214
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (lv)
metrics:
- type: accuracy
value: 40.93140551445864
- type: f1
value: 40.825227889641965
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ml)
metrics:
- type: accuracy
value: 17.88500336247478
- type: f1
value: 17.621569082971817
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (mn)
metrics:
- type: accuracy
value: 32.975790181573636
- type: f1
value: 33.402014633349665
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ms)
metrics:
- type: accuracy
value: 40.91123066577001
- type: f1
value: 40.09538559124075
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (my)
metrics:
- type: accuracy
value: 17.834566240753194
- type: f1
value: 17.006381849454314
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (nb)
metrics:
- type: accuracy
value: 39.47881640887693
- type: f1
value: 37.819934317839305
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (nl)
metrics:
- type: accuracy
value: 41.76193678547412
- type: f1
value: 40.281991759509694
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (pl)
metrics:
- type: accuracy
value: 42.61936785474109
- type: f1
value: 40.83673914649905
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (pt)
metrics:
- type: accuracy
value: 44.54270342972427
- type: f1
value: 43.45243164278448
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ro)
metrics:
- type: accuracy
value: 39.96973772696705
- type: f1
value: 38.74209466530094
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ru)
metrics:
- type: accuracy
value: 37.461331540013454
- type: f1
value: 36.91132021821187
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sl)
metrics:
- type: accuracy
value: 38.28850033624748
- type: f1
value: 37.37259394049676
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sq)
metrics:
- type: accuracy
value: 40.95494283792872
- type: f1
value: 39.767707902869084
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sv)
metrics:
- type: accuracy
value: 41.85272360457296
- type: f1
value: 40.42848260365438
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (sw)
metrics:
- type: accuracy
value: 38.328850033624754
- type: f1
value: 36.90334596675622
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ta)
metrics:
- type: accuracy
value: 19.031607262945528
- type: f1
value: 18.66510306325761
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (te)
metrics:
- type: accuracy
value: 19.38466711499664
- type: f1
value: 19.186399376652535
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (th)
metrics:
- type: accuracy
value: 34.088769334229994
- type: f1
value: 34.20383086009429
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (tl)
metrics:
- type: accuracy
value: 40.285810356422324
- type: f1
value: 39.361500249640414
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (tr)
metrics:
- type: accuracy
value: 38.860121049092136
- type: f1
value: 37.81916859627235
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (ur)
metrics:
- type: accuracy
value: 27.834566240753194
- type: f1
value: 26.898389386106487
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (vi)
metrics:
- type: accuracy
value: 38.70544720914593
- type: f1
value: 38.280026442024415
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (zh-CN)
metrics:
- type: accuracy
value: 45.78009414929387
- type: f1
value: 44.21526778674136
- task:
type: Classification
dataset:
type: mteb/amazon_massive_intent
name: MTEB MassiveIntentClassification (zh-TW)
metrics:
- type: accuracy
value: 42.32010759919301
- type: f1
value: 42.25772977490916
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (af)
metrics:
- type: accuracy
value: 40.24546065904506
- type: f1
value: 38.79924050989544
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (am)
metrics:
- type: accuracy
value: 25.68930733019502
- type: f1
value: 25.488166279162712
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ar)
metrics:
- type: accuracy
value: 32.39744451916611
- type: f1
value: 31.863029579075775
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (az)
metrics:
- type: accuracy
value: 40.53127101546738
- type: f1
value: 39.707079033948936
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (bn)
metrics:
- type: accuracy
value: 27.23268325487559
- type: f1
value: 26.443653281858793
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (cy)
metrics:
- type: accuracy
value: 38.69872225958305
- type: f1
value: 36.55930387892567
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (da)
metrics:
- type: accuracy
value: 44.75453934095494
- type: f1
value: 42.87356484024154
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (de)
metrics:
- type: accuracy
value: 41.355077336919976
- type: f1
value: 39.82365179458047
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (el)
metrics:
- type: accuracy
value: 38.43981170141224
- type: f1
value: 37.02538368296387
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (en)
metrics:
- type: accuracy
value: 66.33826496301278
- type: f1
value: 65.89634765029932
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (es)
metrics:
- type: accuracy
value: 44.17955615332885
- type: f1
value: 43.10228811620319
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (fa)
metrics:
- type: accuracy
value: 34.82851378614661
- type: f1
value: 33.95952441502803
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (fi)
metrics:
- type: accuracy
value: 40.561533288500335
- type: f1
value: 38.04939011733627
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (fr)
metrics:
- type: accuracy
value: 45.917955615332886
- type: f1
value: 44.65741971572902
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (he)
metrics:
- type: accuracy
value: 32.08473436449227
- type: f1
value: 29.53932929808133
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (hi)
metrics:
- type: accuracy
value: 28.369199731002016
- type: f1
value: 27.52902837981212
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (hu)
metrics:
- type: accuracy
value: 39.49226630800269
- type: f1
value: 37.3272340470504
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (hy)
metrics:
- type: accuracy
value: 25.904505716207133
- type: f1
value: 24.547396574853444
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (id)
metrics:
- type: accuracy
value: 40.95830531271016
- type: f1
value: 40.177843177422226
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (is)
metrics:
- type: accuracy
value: 38.564223268325485
- type: f1
value: 37.35307758495248
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (it)
metrics:
- type: accuracy
value: 46.58708809683928
- type: f1
value: 44.103900526804985
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ja)
metrics:
- type: accuracy
value: 46.24747814391393
- type: f1
value: 45.4107101796664
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (jv)
metrics:
- type: accuracy
value: 39.6570275722932
- type: f1
value: 38.82737576832412
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ka)
metrics:
- type: accuracy
value: 25.279085406859448
- type: f1
value: 23.662661686788493
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (km)
metrics:
- type: accuracy
value: 28.97108271687962
- type: f1
value: 27.195758324189246
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (kn)
metrics:
- type: accuracy
value: 19.27370544720915
- type: f1
value: 18.694271924323637
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ko)
metrics:
- type: accuracy
value: 35.729657027572294
- type: f1
value: 34.38287006177308
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (lv)
metrics:
- type: accuracy
value: 39.57296570275723
- type: f1
value: 38.074945140886925
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ml)
metrics:
- type: accuracy
value: 19.895763281775388
- type: f1
value: 20.00931364846829
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (mn)
metrics:
- type: accuracy
value: 32.431069266980494
- type: f1
value: 31.395958664782576
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ms)
metrics:
- type: accuracy
value: 42.32347007397445
- type: f1
value: 40.81374026314701
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (my)
metrics:
- type: accuracy
value: 20.864156018829856
- type: f1
value: 20.409870408935436
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (nb)
metrics:
- type: accuracy
value: 40.47074646940148
- type: f1
value: 39.19044149415904
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (nl)
metrics:
- type: accuracy
value: 43.591123066577
- type: f1
value: 41.43420363064241
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (pl)
metrics:
- type: accuracy
value: 41.876260928043045
- type: f1
value: 41.192117676667614
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (pt)
metrics:
- type: accuracy
value: 46.30800268997983
- type: f1
value: 45.25536730126799
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ro)
metrics:
- type: accuracy
value: 42.525218560860786
- type: f1
value: 41.02418109296485
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ru)
metrics:
- type: accuracy
value: 35.94821788836584
- type: f1
value: 35.08598314806566
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sl)
metrics:
- type: accuracy
value: 38.69199731002017
- type: f1
value: 37.68119408674127
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sq)
metrics:
- type: accuracy
value: 40.474108944182916
- type: f1
value: 39.480530387013594
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sv)
metrics:
- type: accuracy
value: 41.523201075991935
- type: f1
value: 40.20097996024383
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (sw)
metrics:
- type: accuracy
value: 39.54942837928716
- type: f1
value: 38.185561243338064
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ta)
metrics:
- type: accuracy
value: 22.8782784129119
- type: f1
value: 22.239467186721456
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (te)
metrics:
- type: accuracy
value: 20.51445864156019
- type: f1
value: 19.999047885530217
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (th)
metrics:
- type: accuracy
value: 34.92602555480834
- type: f1
value: 33.24016717215723
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (tl)
metrics:
- type: accuracy
value: 40.74983187626093
- type: f1
value: 39.30274328728882
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (tr)
metrics:
- type: accuracy
value: 39.06859448554136
- type: f1
value: 39.21542039662971
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (ur)
metrics:
- type: accuracy
value: 29.747814391392062
- type: f1
value: 28.261836892220447
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (vi)
metrics:
- type: accuracy
value: 38.02286482851379
- type: f1
value: 37.8742438608697
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (zh-CN)
metrics:
- type: accuracy
value: 48.550773369199725
- type: f1
value: 46.7399625882649
- task:
type: Classification
dataset:
type: mteb/amazon_massive_scenario
name: MTEB MassiveScenarioClassification (zh-TW)
metrics:
- type: accuracy
value: 45.17821116341628
- type: f1
value: 44.84809741811729
- task:
type: Clustering
dataset:
type: mteb/medrxiv-clustering-p2p
name: MTEB MedrxivClusteringP2P
metrics:
- type: v_measure
value: 28.301902023313875
- task:
type: Clustering
dataset:
type: mteb/medrxiv-clustering-s2s
name: MTEB MedrxivClusteringS2S
metrics:
- type: v_measure
value: 24.932123582259287
- task:
type: Reranking
dataset:
type: mteb/mind_small
name: MTEB MindSmallReranking
metrics:
- type: map
value: 29.269341041468326
- type: mrr
value: 30.132140876875717
- task:
type: Retrieval
dataset:
type: nfcorpus
name: MTEB NFCorpus
metrics:
- type: map_at_1
value: 1.2269999999999999
- type: map_at_10
value: 3.081
- type: map_at_100
value: 4.104
- type: map_at_1000
value: 4.989
- type: map_at_3
value: 2.221
- type: map_at_5
value: 2.535
- type: ndcg_at_1
value: 15.015
- type: ndcg_at_10
value: 11.805
- type: ndcg_at_100
value: 12.452
- type: ndcg_at_1000
value: 22.284000000000002
- type: ndcg_at_3
value: 13.257
- type: ndcg_at_5
value: 12.199
- type: precision_at_1
value: 16.409000000000002
- type: precision_at_10
value: 9.102
- type: precision_at_100
value: 3.678
- type: precision_at_1000
value: 1.609
- type: precision_at_3
value: 12.797
- type: precision_at_5
value: 10.464
- type: recall_at_1
value: 1.2269999999999999
- type: recall_at_10
value: 5.838
- type: recall_at_100
value: 15.716
- type: recall_at_1000
value: 48.837
- type: recall_at_3
value: 2.828
- type: recall_at_5
value: 3.697
- task:
type: Retrieval
dataset:
type: nq
name: MTEB NQ
metrics:
- type: map_at_1
value: 3.515
- type: map_at_10
value: 5.884
- type: map_at_100
value: 6.510000000000001
- type: map_at_1000
value: 6.598999999999999
- type: map_at_3
value: 4.8919999999999995
- type: map_at_5
value: 5.391
- type: ndcg_at_1
value: 4.056
- type: ndcg_at_10
value: 7.6259999999999994
- type: ndcg_at_100
value: 11.08
- type: ndcg_at_1000
value: 13.793
- type: ndcg_at_3
value: 5.537
- type: ndcg_at_5
value: 6.45
- type: precision_at_1
value: 4.056
- type: precision_at_10
value: 1.4569999999999999
- type: precision_at_100
value: 0.347
- type: precision_at_1000
value: 0.061
- type: precision_at_3
value: 2.6069999999999998
- type: precision_at_5
value: 2.086
- type: recall_at_1
value: 3.515
- type: recall_at_10
value: 12.312
- type: recall_at_100
value: 28.713
- type: recall_at_1000
value: 50.027
- type: recall_at_3
value: 6.701
- type: recall_at_5
value: 8.816
- task:
type: Retrieval
dataset:
type: quora
name: MTEB QuoraRetrieval
metrics:
- type: map_at_1
value: 61.697
- type: map_at_10
value: 74.20400000000001
- type: map_at_100
value: 75.023
- type: map_at_1000
value: 75.059
- type: map_at_3
value: 71.265
- type: map_at_5
value: 73.001
- type: ndcg_at_1
value: 70.95
- type: ndcg_at_10
value: 78.96
- type: ndcg_at_100
value: 81.26
- type: ndcg_at_1000
value: 81.679
- type: ndcg_at_3
value: 75.246
- type: ndcg_at_5
value: 77.092
- type: precision_at_1
value: 70.95
- type: precision_at_10
value: 11.998000000000001
- type: precision_at_100
value: 1.451
- type: precision_at_1000
value: 0.154
- type: precision_at_3
value: 32.629999999999995
- type: precision_at_5
value: 21.573999999999998
- type: recall_at_1
value: 61.697
- type: recall_at_10
value: 88.23299999999999
- type: recall_at_100
value: 96.961
- type: recall_at_1000
value: 99.401
- type: recall_at_3
value: 77.689
- type: recall_at_5
value: 82.745
- task:
type: Clustering
dataset:
type: mteb/reddit-clustering
name: MTEB RedditClustering
metrics:
- type: v_measure
value: 33.75741018380938
- task:
type: Clustering
dataset:
type: mteb/reddit-clustering-p2p
name: MTEB RedditClusteringP2P
metrics:
- type: v_measure
value: 41.00799910099266
- task:
type: Retrieval
dataset:
type: scidocs
name: MTEB SCIDOCS
metrics:
- type: map_at_1
value: 1.72
- type: map_at_10
value: 3.8240000000000003
- type: map_at_100
value: 4.727
- type: map_at_1000
value: 4.932
- type: map_at_3
value: 2.867
- type: map_at_5
value: 3.3230000000000004
- type: ndcg_at_1
value: 8.5
- type: ndcg_at_10
value: 7.133000000000001
- type: ndcg_at_100
value: 11.911
- type: ndcg_at_1000
value: 16.962
- type: ndcg_at_3
value: 6.763
- type: ndcg_at_5
value: 5.832
- type: precision_at_1
value: 8.5
- type: precision_at_10
value: 3.6799999999999997
- type: precision_at_100
value: 1.0670000000000002
- type: precision_at_1000
value: 0.22999999999999998
- type: precision_at_3
value: 6.2330000000000005
- type: precision_at_5
value: 5.0200000000000005
- type: recall_at_1
value: 1.72
- type: recall_at_10
value: 7.487000000000001
- type: recall_at_100
value: 21.683
- type: recall_at_1000
value: 46.688
- type: recall_at_3
value: 3.798
- type: recall_at_5
value: 5.113
- task:
type: STS
dataset:
type: mteb/sickr-sts
name: MTEB SICK-R
metrics:
- type: cos_sim_pearson
value: 80.96286245858941
- type: cos_sim_spearman
value: 74.57093488947429
- type: euclidean_pearson
value: 75.50377970259402
- type: euclidean_spearman
value: 71.7498004622999
- type: manhattan_pearson
value: 75.3256836091382
- type: manhattan_spearman
value: 71.80676733410375
- task:
type: STS
dataset:
type: mteb/sts12-sts
name: MTEB STS12
metrics:
- type: cos_sim_pearson
value: 80.20938796088339
- type: cos_sim_spearman
value: 69.16914010333394
- type: euclidean_pearson
value: 79.33415250097545
- type: euclidean_spearman
value: 71.46707320292745
- type: manhattan_pearson
value: 79.73669837981976
- type: manhattan_spearman
value: 71.87919511134902
- task:
type: STS
dataset:
type: mteb/sts13-sts
name: MTEB STS13
metrics:
- type: cos_sim_pearson
value: 76.401935081936
- type: cos_sim_spearman
value: 77.23446219694267
- type: euclidean_pearson
value: 74.61017160439877
- type: euclidean_spearman
value: 75.85871531365609
- type: manhattan_pearson
value: 74.83034779539724
- type: manhattan_spearman
value: 75.95948993588429
- task:
type: STS
dataset:
type: mteb/sts14-sts
name: MTEB STS14
metrics:
- type: cos_sim_pearson
value: 75.35551963935667
- type: cos_sim_spearman
value: 70.98892671568665
- type: euclidean_pearson
value: 73.24467338564628
- type: euclidean_spearman
value: 71.97533151639425
- type: manhattan_pearson
value: 73.2776559359938
- type: manhattan_spearman
value: 72.2221421456084
- task:
type: STS
dataset:
type: mteb/sts15-sts
name: MTEB STS15
metrics:
- type: cos_sim_pearson
value: 79.05293131911803
- type: cos_sim_spearman
value: 79.7379478259805
- type: euclidean_pearson
value: 78.17016171851057
- type: euclidean_spearman
value: 78.76038607583105
- type: manhattan_pearson
value: 78.4994607532332
- type: manhattan_spearman
value: 79.13026720132872
- task:
type: STS
dataset:
type: mteb/sts16-sts
name: MTEB STS16
metrics:
- type: cos_sim_pearson
value: 76.04750373932828
- type: cos_sim_spearman
value: 77.93230986462234
- type: euclidean_pearson
value: 75.8320302521164
- type: euclidean_spearman
value: 76.83154481579385
- type: manhattan_pearson
value: 75.98713517720608
- type: manhattan_spearman
value: 76.95479705521507
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (ko-ko)
metrics:
- type: cos_sim_pearson
value: 43.0464619152799
- type: cos_sim_spearman
value: 45.65606588928089
- type: euclidean_pearson
value: 45.69437788355499
- type: euclidean_spearman
value: 45.08552742346606
- type: manhattan_pearson
value: 45.87166698903681
- type: manhattan_spearman
value: 45.155963016434164
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (ar-ar)
metrics:
- type: cos_sim_pearson
value: 53.27469278912148
- type: cos_sim_spearman
value: 54.16113207623789
- type: euclidean_pearson
value: 55.97026429327157
- type: euclidean_spearman
value: 54.71320909074608
- type: manhattan_pearson
value: 56.12511774278802
- type: manhattan_spearman
value: 55.22875659158676
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-ar)
metrics:
- type: cos_sim_pearson
value: 1.5482997790039945
- type: cos_sim_spearman
value: 1.7208386347363582
- type: euclidean_pearson
value: 6.727915670345885
- type: euclidean_spearman
value: 6.112826908474543
- type: manhattan_pearson
value: 4.94386093060865
- type: manhattan_spearman
value: 5.018174110623732
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-de)
metrics:
- type: cos_sim_pearson
value: 27.5420218362265
- type: cos_sim_spearman
value: 25.483838431031007
- type: euclidean_pearson
value: 6.268684143856358
- type: euclidean_spearman
value: 5.877961421091679
- type: manhattan_pearson
value: 2.667237739227861
- type: manhattan_spearman
value: 2.5683839956554775
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-en)
metrics:
- type: cos_sim_pearson
value: 85.32029757646663
- type: cos_sim_spearman
value: 87.32720847297225
- type: euclidean_pearson
value: 81.12594485791254
- type: euclidean_spearman
value: 81.1531079489332
- type: manhattan_pearson
value: 81.32899414704019
- type: manhattan_spearman
value: 81.3897040261192
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (en-tr)
metrics:
- type: cos_sim_pearson
value: 4.37162299241808
- type: cos_sim_spearman
value: 2.0879072561774543
- type: euclidean_pearson
value: 3.0725243785454595
- type: euclidean_spearman
value: 5.3721339279483535
- type: manhattan_pearson
value: 4.867795293367359
- type: manhattan_spearman
value: 7.9397069840018775
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (es-en)
metrics:
- type: cos_sim_pearson
value: 20.306030448858603
- type: cos_sim_spearman
value: 21.93220782551375
- type: euclidean_pearson
value: 3.878631934602361
- type: euclidean_spearman
value: 5.171796902725965
- type: manhattan_pearson
value: 7.13020644036815
- type: manhattan_spearman
value: 7.707315591498748
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (es-es)
metrics:
- type: cos_sim_pearson
value: 66.81873207478459
- type: cos_sim_spearman
value: 67.80273445636502
- type: euclidean_pearson
value: 70.60654682977268
- type: euclidean_spearman
value: 69.4566208379486
- type: manhattan_pearson
value: 70.9548461896642
- type: manhattan_spearman
value: 69.78323323058773
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (fr-en)
metrics:
- type: cos_sim_pearson
value: 21.366487281202602
- type: cos_sim_spearman
value: 18.90627528698481
- type: euclidean_pearson
value: 2.3390998579461995
- type: euclidean_spearman
value: 4.151213674012541
- type: manhattan_pearson
value: 2.234831868844863
- type: manhattan_spearman
value: 4.555291328501442
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (it-en)
metrics:
- type: cos_sim_pearson
value: 20.73153177251085
- type: cos_sim_spearman
value: 16.3855949033176
- type: euclidean_pearson
value: 8.734648741714238
- type: euclidean_spearman
value: 10.75672244732182
- type: manhattan_pearson
value: 7.536654126608877
- type: manhattan_spearman
value: 8.330065460047296
- task:
type: STS
dataset:
type: mteb/sts17-crosslingual-sts
name: MTEB STS17 (nl-en)
metrics:
- type: cos_sim_pearson
value: 26.618435024084253
- type: cos_sim_spearman
value: 23.488974089577816
- type: euclidean_pearson
value: 3.1310350304707866
- type: euclidean_spearman
value: 3.1242598481634665
- type: manhattan_pearson
value: 1.1096752982707008
- type: manhattan_spearman
value: 1.4591693078765848
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (en)
metrics:
- type: cos_sim_pearson
value: 59.17638344661753
- type: cos_sim_spearman
value: 59.636760071130865
- type: euclidean_pearson
value: 56.68753290255448
- type: euclidean_spearman
value: 57.613280258574484
- type: manhattan_pearson
value: 56.92312052723706
- type: manhattan_spearman
value: 57.76774918418505
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de)
metrics:
- type: cos_sim_pearson
value: 10.322254716987457
- type: cos_sim_spearman
value: 11.0033092996862
- type: euclidean_pearson
value: 6.006926471684402
- type: euclidean_spearman
value: 10.972140246688376
- type: manhattan_pearson
value: 5.933298751861177
- type: manhattan_spearman
value: 11.030111585680233
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (es)
metrics:
- type: cos_sim_pearson
value: 43.38031880545056
- type: cos_sim_spearman
value: 43.05358201410913
- type: euclidean_pearson
value: 42.72327196362553
- type: euclidean_spearman
value: 42.55163899944477
- type: manhattan_pearson
value: 44.01557499780587
- type: manhattan_spearman
value: 43.12473221615855
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (pl)
metrics:
- type: cos_sim_pearson
value: 4.291290504363136
- type: cos_sim_spearman
value: 14.912727487893479
- type: euclidean_pearson
value: 3.2855132112394485
- type: euclidean_spearman
value: 16.575204463951025
- type: manhattan_pearson
value: 3.2398776723465814
- type: manhattan_spearman
value: 16.841985772913855
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (tr)
metrics:
- type: cos_sim_pearson
value: 4.102739498555817
- type: cos_sim_spearman
value: 3.818238576547375
- type: euclidean_pearson
value: 2.3181033496453556
- type: euclidean_spearman
value: 5.1826811802703565
- type: manhattan_pearson
value: 4.8006179265256455
- type: manhattan_spearman
value: 6.738401400306252
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (ar)
metrics:
- type: cos_sim_pearson
value: 2.38765395226737
- type: cos_sim_spearman
value: 5.173899391162327
- type: euclidean_pearson
value: 3.0710263954769825
- type: euclidean_spearman
value: 5.04922290903982
- type: manhattan_pearson
value: 3.7826314109861703
- type: manhattan_spearman
value: 5.042238232170212
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (ru)
metrics:
- type: cos_sim_pearson
value: 7.6735490672676345
- type: cos_sim_spearman
value: 3.3631215256878892
- type: euclidean_pearson
value: 4.64331702652217
- type: euclidean_spearman
value: 3.6129205171334324
- type: manhattan_pearson
value: 4.011231736076196
- type: manhattan_spearman
value: 3.233959766173701
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (zh)
metrics:
- type: cos_sim_pearson
value: 0.06167614416104335
- type: cos_sim_spearman
value: 6.521685391703255
- type: euclidean_pearson
value: 4.884572579069032
- type: euclidean_spearman
value: 5.59058032900239
- type: manhattan_pearson
value: 6.139838096573897
- type: manhattan_spearman
value: 5.0060884837066215
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (fr)
metrics:
- type: cos_sim_pearson
value: 53.19490347682836
- type: cos_sim_spearman
value: 54.56055727079527
- type: euclidean_pearson
value: 52.55574442039842
- type: euclidean_spearman
value: 52.94640154371587
- type: manhattan_pearson
value: 53.275993040454196
- type: manhattan_spearman
value: 53.174561503510155
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de-en)
metrics:
- type: cos_sim_pearson
value: 51.151158530122146
- type: cos_sim_spearman
value: 53.926925081736655
- type: euclidean_pearson
value: 44.55629287737235
- type: euclidean_spearman
value: 46.222372143731384
- type: manhattan_pearson
value: 42.831322151459005
- type: manhattan_spearman
value: 45.70991764985799
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (es-en)
metrics:
- type: cos_sim_pearson
value: 30.36194885126792
- type: cos_sim_spearman
value: 32.739632941633836
- type: euclidean_pearson
value: 29.83135800843496
- type: euclidean_spearman
value: 31.114406001326923
- type: manhattan_pearson
value: 31.264502938148286
- type: manhattan_spearman
value: 33.3112040753475
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (it)
metrics:
- type: cos_sim_pearson
value: 35.23883630335275
- type: cos_sim_spearman
value: 33.67797082086704
- type: euclidean_pearson
value: 34.878640693874544
- type: euclidean_spearman
value: 33.525189235133496
- type: manhattan_pearson
value: 34.22761246389947
- type: manhattan_spearman
value: 32.713218497609176
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (pl-en)
metrics:
- type: cos_sim_pearson
value: 19.809302548119547
- type: cos_sim_spearman
value: 20.540370202115497
- type: euclidean_pearson
value: 23.006803962133016
- type: euclidean_spearman
value: 22.96270653079511
- type: manhattan_pearson
value: 25.40168317585851
- type: manhattan_spearman
value: 25.421508137540865
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (zh-en)
metrics:
- type: cos_sim_pearson
value: 20.393500955410488
- type: cos_sim_spearman
value: 26.705713693011603
- type: euclidean_pearson
value: 18.168376767724585
- type: euclidean_spearman
value: 19.260826601517245
- type: manhattan_pearson
value: 18.302619990671527
- type: manhattan_spearman
value: 19.4691037846159
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (es-it)
metrics:
- type: cos_sim_pearson
value: 36.58919983075148
- type: cos_sim_spearman
value: 35.989722099974045
- type: euclidean_pearson
value: 41.045112547574206
- type: euclidean_spearman
value: 39.322301680629835
- type: manhattan_pearson
value: 41.36802503205308
- type: manhattan_spearman
value: 40.76270030293609
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de-fr)
metrics:
- type: cos_sim_pearson
value: 26.350936227950083
- type: cos_sim_spearman
value: 25.108218032460343
- type: euclidean_pearson
value: 28.61681094744849
- type: euclidean_spearman
value: 27.350990203943592
- type: manhattan_pearson
value: 30.527977072984513
- type: manhattan_spearman
value: 26.403339990640813
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (de-pl)
metrics:
- type: cos_sim_pearson
value: 20.056269198600322
- type: cos_sim_spearman
value: 20.939990379746757
- type: euclidean_pearson
value: 18.942765438962198
- type: euclidean_spearman
value: 21.709842967237446
- type: manhattan_pearson
value: 23.643909798655123
- type: manhattan_spearman
value: 23.58828328071473
- task:
type: STS
dataset:
type: mteb/sts22-crosslingual-sts
name: MTEB STS22 (fr-pl)
metrics:
- type: cos_sim_pearson
value: 19.563740271419395
- type: cos_sim_spearman
value: 5.634361698190111
- type: euclidean_pearson
value: 16.833522619239474
- type: euclidean_spearman
value: 16.903085094570333
- type: manhattan_pearson
value: 5.805392712660814
- type: manhattan_spearman
value: 16.903085094570333
- task:
type: STS
dataset:
type: mteb/stsbenchmark-sts
name: MTEB STSBenchmark
metrics:
- type: cos_sim_pearson
value: 80.00905671833966
- type: cos_sim_spearman
value: 79.54269211027272
- type: euclidean_pearson
value: 79.51954544247441
- type: euclidean_spearman
value: 78.93670303434288
- type: manhattan_pearson
value: 79.47610653340678
- type: manhattan_spearman
value: 79.07344156719613
- task:
type: Reranking
dataset:
type: mteb/scidocs-reranking
name: MTEB SciDocsRR
metrics:
- type: map
value: 68.35710819755543
- type: mrr
value: 88.05442832403617
- task:
type: Retrieval
dataset:
type: scifact
name: MTEB SciFact
metrics:
- type: map_at_1
value: 21.556
- type: map_at_10
value: 27.982000000000003
- type: map_at_100
value: 28.937
- type: map_at_1000
value: 29.058
- type: map_at_3
value: 25.644
- type: map_at_5
value: 26.996
- type: ndcg_at_1
value: 23.333000000000002
- type: ndcg_at_10
value: 31.787
- type: ndcg_at_100
value: 36.647999999999996
- type: ndcg_at_1000
value: 39.936
- type: ndcg_at_3
value: 27.299
- type: ndcg_at_5
value: 29.659000000000002
- type: precision_at_1
value: 23.333000000000002
- type: precision_at_10
value: 4.867
- type: precision_at_100
value: 0.743
- type: precision_at_1000
value: 0.10200000000000001
- type: precision_at_3
value: 11.333
- type: precision_at_5
value: 8.133
- type: recall_at_1
value: 21.556
- type: recall_at_10
value: 42.333
- type: recall_at_100
value: 65.706
- type: recall_at_1000
value: 91.489
- type: recall_at_3
value: 30.361
- type: recall_at_5
value: 36.222
- task:
type: PairClassification
dataset:
type: mteb/sprintduplicatequestions-pairclassification
name: MTEB SprintDuplicateQuestions
metrics:
- type: cos_sim_accuracy
value: 99.49306930693069
- type: cos_sim_ap
value: 77.7308550291728
- type: cos_sim_f1
value: 71.78978681209718
- type: cos_sim_precision
value: 71.1897738446411
- type: cos_sim_recall
value: 72.39999999999999
- type: dot_accuracy
value: 99.08118811881188
- type: dot_ap
value: 30.267748833368234
- type: dot_f1
value: 34.335201222618444
- type: dot_precision
value: 34.994807892004154
- type: dot_recall
value: 33.7
- type: euclidean_accuracy
value: 99.51683168316832
- type: euclidean_ap
value: 78.64498778235628
- type: euclidean_f1
value: 73.09149972929075
- type: euclidean_precision
value: 79.69303423848878
- type: euclidean_recall
value: 67.5
- type: manhattan_accuracy
value: 99.53168316831683
- type: manhattan_ap
value: 79.45274878693958
- type: manhattan_f1
value: 74.19863373620599
- type: manhattan_precision
value: 78.18383167220377
- type: manhattan_recall
value: 70.6
- type: max_accuracy
value: 99.53168316831683
- type: max_ap
value: 79.45274878693958
- type: max_f1
value: 74.19863373620599
- task:
type: Clustering
dataset:
type: mteb/stackexchange-clustering
name: MTEB StackExchangeClustering
metrics:
- type: v_measure
value: 44.59127540530939
- task:
type: Clustering
dataset:
type: mteb/stackexchange-clustering-p2p
name: MTEB StackExchangeClusteringP2P
metrics:
- type: v_measure
value: 28.230204578753636
- task:
type: Reranking
dataset:
type: mteb/stackoverflowdupquestions-reranking
name: MTEB StackOverflowDupQuestions
metrics:
- type: map
value: 39.96520488022785
- type: mrr
value: 40.189248047703934
- task:
type: Summarization
dataset:
type: mteb/summeval
name: MTEB SummEval
metrics:
- type: cos_sim_pearson
value: 30.56303767714449
- type: cos_sim_spearman
value: 30.256847004390487
- type: dot_pearson
value: 29.453520030995005
- type: dot_spearman
value: 29.561732550926777
- task:
type: Retrieval
dataset:
type: trec-covid
name: MTEB TRECCOVID
metrics:
- type: map_at_1
value: 0.11299999999999999
- type: map_at_10
value: 0.733
- type: map_at_100
value: 3.313
- type: map_at_1000
value: 7.355
- type: map_at_3
value: 0.28200000000000003
- type: map_at_5
value: 0.414
- type: ndcg_at_1
value: 42.0
- type: ndcg_at_10
value: 39.31
- type: ndcg_at_100
value: 26.904
- type: ndcg_at_1000
value: 23.778
- type: ndcg_at_3
value: 42.775999999999996
- type: ndcg_at_5
value: 41.554
- type: precision_at_1
value: 48.0
- type: precision_at_10
value: 43.0
- type: precision_at_100
value: 27.08
- type: precision_at_1000
value: 11.014
- type: precision_at_3
value: 48.0
- type: precision_at_5
value: 45.6
- type: recall_at_1
value: 0.11299999999999999
- type: recall_at_10
value: 0.976
- type: recall_at_100
value: 5.888
- type: recall_at_1000
value: 22.634999999999998
- type: recall_at_3
value: 0.329
- type: recall_at_5
value: 0.518
- task:
type: Retrieval
dataset:
type: webis-touche2020
name: MTEB Touche2020
metrics:
- type: map_at_1
value: 0.645
- type: map_at_10
value: 4.1160000000000005
- type: map_at_100
value: 7.527
- type: map_at_1000
value: 8.677999999999999
- type: map_at_3
value: 1.6019999999999999
- type: map_at_5
value: 2.6
- type: ndcg_at_1
value: 10.204
- type: ndcg_at_10
value: 12.27
- type: ndcg_at_100
value: 22.461000000000002
- type: ndcg_at_1000
value: 33.543
- type: ndcg_at_3
value: 9.982000000000001
- type: ndcg_at_5
value: 11.498
- type: precision_at_1
value: 10.204
- type: precision_at_10
value: 12.245000000000001
- type: precision_at_100
value: 5.286
- type: precision_at_1000
value: 1.2630000000000001
- type: precision_at_3
value: 10.884
- type: precision_at_5
value: 13.061
- type: recall_at_1
value: 0.645
- type: recall_at_10
value: 8.996
- type: recall_at_100
value: 33.666000000000004
- type: recall_at_1000
value: 67.704
- type: recall_at_3
value: 2.504
- type: recall_at_5
value: 4.95
- task:
type: Classification
dataset:
type: mteb/toxic_conversations_50k
name: MTEB ToxicConversationsClassification
metrics:
- type: accuracy
value: 62.7862
- type: ap
value: 10.958454618347831
- type: f1
value: 48.37243417046763
- task:
type: Classification
dataset:
type: mteb/tweet_sentiment_extraction
name: MTEB TweetSentimentExtractionClassification
metrics:
- type: accuracy
value: 54.821731748726656
- type: f1
value: 55.14729314789282
- task:
type: Clustering
dataset:
type: mteb/twentynewsgroups-clustering
name: MTEB TwentyNewsgroupsClustering
metrics:
- type: v_measure
value: 28.24295128553035
- task:
type: PairClassification
dataset:
type: mteb/twittersemeval2015-pairclassification
name: MTEB TwitterSemEval2015
metrics:
- type: cos_sim_accuracy
value: 81.5640460153782
- type: cos_sim_ap
value: 57.094095366921536
- type: cos_sim_f1
value: 55.29607083563918
- type: cos_sim_precision
value: 47.62631077216397
- type: cos_sim_recall
value: 65.91029023746702
- type: dot_accuracy
value: 78.81623651427549
- type: dot_ap
value: 47.42989400382077
- type: dot_f1
value: 51.25944584382871
- type: dot_precision
value: 42.55838271174625
- type: dot_recall
value: 64.43271767810026
- type: euclidean_accuracy
value: 80.29445073612685
- type: euclidean_ap
value: 53.42012231336148
- type: euclidean_f1
value: 51.867783563504645
- type: euclidean_precision
value: 45.4203013481364
- type: euclidean_recall
value: 60.4485488126649
- type: manhattan_accuracy
value: 80.2884901949097
- type: manhattan_ap
value: 53.43205271323232
- type: manhattan_f1
value: 52.014165559982295
- type: manhattan_precision
value: 44.796035074342356
- type: manhattan_recall
value: 62.00527704485488
- type: max_accuracy
value: 81.5640460153782
- type: max_ap
value: 57.094095366921536
- type: max_f1
value: 55.29607083563918
- task:
type: PairClassification
dataset:
type: mteb/twitterurlcorpus-pairclassification
name: MTEB TwitterURLCorpus
metrics:
- type: cos_sim_accuracy
value: 86.63018589668955
- type: cos_sim_ap
value: 80.51063771262909
- type: cos_sim_f1
value: 72.70810586950793
- type: cos_sim_precision
value: 71.14123627790467
- type: cos_sim_recall
value: 74.3455497382199
- type: dot_accuracy
value: 82.41743315092948
- type: dot_ap
value: 69.2393381283664
- type: dot_f1
value: 65.61346624814597
- type: dot_precision
value: 59.43260638630257
- type: dot_recall
value: 73.22913458577148
- type: euclidean_accuracy
value: 86.49435324251951
- type: euclidean_ap
value: 80.28100477250926
- type: euclidean_f1
value: 72.58242344489099
- type: euclidean_precision
value: 67.44662568576906
- type: euclidean_recall
value: 78.56482907299045
- type: manhattan_accuracy
value: 86.59525749990297
- type: manhattan_ap
value: 80.37850832566262
- type: manhattan_f1
value: 72.59435321233073
- type: manhattan_precision
value: 68.19350473612991
- type: manhattan_recall
value: 77.60240221743148
- type: max_accuracy
value: 86.63018589668955
- type: max_ap
value: 80.51063771262909
- type: max_f1
value: 72.70810586950793
---
# SGPT-125M-weightedmean-nli-bitfit
## Usage
For usage instructions, refer to our codebase: https://github.com/Muennighoff/sgpt
## Evaluation Results
For eval results, refer to the eval folder or our paper: https://arxiv.org/abs/2202.08904
## Training
The model was trained with the parameters:
**DataLoader**:
`sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader` of length 8807 with parameters:
```
{'batch_size': 64}
```
**Loss**:
`sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss` with parameters:
```
{'scale': 20.0, 'similarity_fct': 'cos_sim'}
```
Parameters of the fit()-Method:
```
{
"epochs": 1,
"evaluation_steps": 880,
"evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
"max_grad_norm": 1,
"optimizer_class": "<class 'transformers.optimization.AdamW'>",
"optimizer_params": {
"lr": 0.0002
},
"scheduler": "WarmupLinear",
"steps_per_epoch": null,
"warmup_steps": 881,
"weight_decay": 0.01
}
```
## Full Model Architecture
```
SentenceTransformer(
(0): Transformer({'max_seq_length': 75, 'do_lower_case': False}) with Transformer model: GPTNeoModel
(1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': True, 'pooling_mode_lasttoken': False})
)
```
## Citing & Authors
```bibtex
@article{muennighoff2022sgpt,
title={SGPT: GPT Sentence Embeddings for Semantic Search},
author={Muennighoff, Niklas},
journal={arXiv preprint arXiv:2202.08904},
year={2022}
}
```