diff --git "a/README.md" "b/README.md" new file mode 100644--- /dev/null +++ "b/README.md" @@ -0,0 +1,6061 @@ +--- +tags: +- mteb +model-index: +- name: multilingual-e5-small + results: + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (en) + config: en + split: test + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + metrics: + - type: accuracy + value: 73.79104477611939 + - type: ap + value: 36.9996434842022 + - type: f1 + value: 67.95453679103099 + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (de) + config: de + split: test + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + metrics: + - type: accuracy + value: 71.64882226980728 + - type: ap + value: 82.11942130026586 + - type: f1 + value: 69.87963421606715 + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (en-ext) + config: en-ext + split: test + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + metrics: + - type: accuracy + value: 75.8095952023988 + - type: ap + value: 24.46869495579561 + - type: f1 + value: 63.00108480037597 + - task: + type: Classification + dataset: + type: mteb/amazon_counterfactual + name: MTEB AmazonCounterfactualClassification (ja) + config: ja + split: test + revision: e8379541af4e31359cca9fbcf4b00f2671dba205 + metrics: + - type: accuracy + value: 64.186295503212 + - type: ap + value: 15.496804690197042 + - type: f1 + value: 52.07153895475031 + - task: + type: Classification + dataset: + type: mteb/amazon_polarity + name: MTEB AmazonPolarityClassification + config: default + split: test + revision: e2d317d38cd51312af73b3d32a06d1a08b442046 + metrics: + - type: accuracy + value: 88.699325 + - type: ap + value: 85.27039559917269 + - type: f1 + value: 88.65556295032513 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (en) + config: en + split: test + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + metrics: + - type: accuracy + value: 44.69799999999999 + - type: f1 + value: 43.73187348654165 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (de) + config: de + split: test + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + metrics: + - type: accuracy + value: 40.245999999999995 + - type: f1 + value: 39.3863530637684 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (es) + config: es + split: test + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + metrics: + - type: accuracy + value: 40.394 + - type: f1 + value: 39.301223469483446 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (fr) + config: fr + split: test + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + metrics: + - type: accuracy + value: 38.864 + - type: f1 + value: 37.97974261868003 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (ja) + config: ja + split: test + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + metrics: + - type: accuracy + value: 37.682 + - type: f1 + value: 37.07399369768313 + - task: + type: Classification + dataset: + type: mteb/amazon_reviews_multi + name: MTEB AmazonReviewsClassification (zh) + config: zh + split: test + revision: 1399c76144fd37290681b995c656ef9b2e06e26d + metrics: + - type: accuracy + value: 37.504 + - type: f1 + value: 36.62317273874278 + - task: + type: Retrieval + dataset: + type: arguana + name: MTEB ArguAna + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 19.061 + - type: map_at_10 + value: 31.703 + - type: map_at_100 + value: 32.967 + - type: map_at_1000 + value: 33.001000000000005 + - type: map_at_3 + value: 27.466 + - type: map_at_5 + value: 29.564 + - type: mrr_at_1 + value: 19.559 + - type: mrr_at_10 + value: 31.874999999999996 + - type: mrr_at_100 + value: 33.146 + - type: mrr_at_1000 + value: 33.18 + - type: mrr_at_3 + value: 27.667 + - type: mrr_at_5 + value: 29.74 + - type: ndcg_at_1 + value: 19.061 + - type: ndcg_at_10 + value: 39.062999999999995 + - type: ndcg_at_100 + value: 45.184000000000005 + - type: ndcg_at_1000 + value: 46.115 + - type: ndcg_at_3 + value: 30.203000000000003 + - type: ndcg_at_5 + value: 33.953 + - type: precision_at_1 + value: 19.061 + - type: precision_at_10 + value: 6.279999999999999 + - type: precision_at_100 + value: 0.9129999999999999 + - type: precision_at_1000 + value: 0.099 + - type: precision_at_3 + value: 12.706999999999999 + - type: precision_at_5 + value: 9.431000000000001 + - type: recall_at_1 + value: 19.061 + - type: recall_at_10 + value: 62.802 + - type: recall_at_100 + value: 91.323 + - type: recall_at_1000 + value: 98.72 + - type: recall_at_3 + value: 38.122 + - type: recall_at_5 + value: 47.155 + - task: + type: Clustering + dataset: + type: mteb/arxiv-clustering-p2p + name: MTEB ArxivClusteringP2P + config: default + split: test + revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d + metrics: + - type: v_measure + value: 39.22266660528253 + - task: + type: Clustering + dataset: + type: mteb/arxiv-clustering-s2s + name: MTEB ArxivClusteringS2S + config: default + split: test + revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 + metrics: + - type: v_measure + value: 30.79980849482483 + - task: + type: Reranking + dataset: + type: mteb/askubuntudupquestions-reranking + name: MTEB AskUbuntuDupQuestions + config: default + split: test + revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 + metrics: + - type: map + value: 57.8790068352054 + - type: mrr + value: 71.78791276436706 + - task: + type: STS + dataset: + type: mteb/biosses-sts + name: MTEB BIOSSES + config: default + split: test + revision: d3fb88f8f02e40887cd149695127462bbcf29b4a + metrics: + - type: cos_sim_pearson + value: 82.36328364043163 + - type: cos_sim_spearman + value: 82.26211536195868 + - type: euclidean_pearson + value: 80.3183865039173 + - type: euclidean_spearman + value: 79.88495276296132 + - type: manhattan_pearson + value: 80.14484480692127 + - type: manhattan_spearman + value: 80.39279565980743 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (de-en) + config: de-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 98.0375782881002 + - type: f1 + value: 97.86012526096033 + - type: precision + value: 97.77139874739039 + - type: recall + value: 98.0375782881002 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (fr-en) + config: fr-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 93.35241030156286 + - type: f1 + value: 92.66050333846944 + - type: precision + value: 92.3306919069631 + - type: recall + value: 93.35241030156286 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (ru-en) + config: ru-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 94.0699688257707 + - type: f1 + value: 93.50236693222492 + - type: precision + value: 93.22791825424315 + - type: recall + value: 94.0699688257707 + - task: + type: BitextMining + dataset: + type: mteb/bucc-bitext-mining + name: MTEB BUCC (zh-en) + config: zh-en + split: test + revision: d51519689f32196a32af33b075a01d0e7c51e252 + metrics: + - type: accuracy + value: 89.25750394944708 + - type: f1 + value: 88.79234684921889 + - type: precision + value: 88.57293312269616 + - type: recall + value: 89.25750394944708 + - task: + type: Classification + dataset: + type: mteb/banking77 + name: MTEB Banking77Classification + config: default + split: test + revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 + metrics: + - type: accuracy + value: 79.41558441558442 + - type: f1 + value: 79.25886487487219 + - task: + type: Clustering + dataset: + type: mteb/biorxiv-clustering-p2p + name: MTEB BiorxivClusteringP2P + config: default + split: test + revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 + metrics: + - type: v_measure + value: 35.747820820329736 + - task: + type: Clustering + dataset: + type: mteb/biorxiv-clustering-s2s + name: MTEB BiorxivClusteringS2S + config: default + split: test + revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 + metrics: + - type: v_measure + value: 27.045143830596146 + - task: + type: Retrieval + dataset: + type: BeIR/cqadupstack + name: MTEB CQADupstackRetrieval + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 24.252999999999997 + - type: map_at_10 + value: 31.655916666666666 + - type: map_at_100 + value: 32.680749999999996 + - type: map_at_1000 + value: 32.79483333333334 + - type: map_at_3 + value: 29.43691666666666 + - type: map_at_5 + value: 30.717416666666665 + - type: mrr_at_1 + value: 28.602750000000004 + - type: mrr_at_10 + value: 35.56875 + - type: mrr_at_100 + value: 36.3595 + - type: mrr_at_1000 + value: 36.427749999999996 + - type: mrr_at_3 + value: 33.586166666666664 + - type: mrr_at_5 + value: 34.73641666666666 + - type: ndcg_at_1 + value: 28.602750000000004 + - type: ndcg_at_10 + value: 36.06933333333334 + - type: ndcg_at_100 + value: 40.70141666666667 + - type: ndcg_at_1000 + value: 43.24341666666667 + - type: ndcg_at_3 + value: 32.307916666666664 + - type: ndcg_at_5 + value: 34.129999999999995 + - type: precision_at_1 + value: 28.602750000000004 + - type: precision_at_10 + value: 6.097666666666667 + - type: precision_at_100 + value: 0.9809166666666668 + - type: precision_at_1000 + value: 0.13766666666666663 + - type: precision_at_3 + value: 14.628166666666667 + - type: precision_at_5 + value: 10.266916666666667 + - type: recall_at_1 + value: 24.252999999999997 + - type: recall_at_10 + value: 45.31916666666667 + - type: recall_at_100 + value: 66.03575000000001 + - type: recall_at_1000 + value: 83.94708333333334 + - type: recall_at_3 + value: 34.71941666666666 + - type: recall_at_5 + value: 39.46358333333333 + - task: + type: Retrieval + dataset: + type: climate-fever + name: MTEB ClimateFEVER + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 9.024000000000001 + - type: map_at_10 + value: 15.644 + - type: map_at_100 + value: 17.154 + - type: map_at_1000 + value: 17.345 + - type: map_at_3 + value: 13.028 + - type: map_at_5 + value: 14.251 + - type: mrr_at_1 + value: 19.674 + - type: mrr_at_10 + value: 29.826999999999998 + - type: mrr_at_100 + value: 30.935000000000002 + - type: mrr_at_1000 + value: 30.987 + - type: mrr_at_3 + value: 26.645000000000003 + - type: mrr_at_5 + value: 28.29 + - type: ndcg_at_1 + value: 19.674 + - type: ndcg_at_10 + value: 22.545 + - type: ndcg_at_100 + value: 29.207 + - type: ndcg_at_1000 + value: 32.912 + - type: ndcg_at_3 + value: 17.952 + - type: ndcg_at_5 + value: 19.363 + - type: precision_at_1 + value: 19.674 + - type: precision_at_10 + value: 7.212000000000001 + - type: precision_at_100 + value: 1.435 + - type: precision_at_1000 + value: 0.212 + - type: precision_at_3 + value: 13.507 + - type: precision_at_5 + value: 10.397 + - type: recall_at_1 + value: 9.024000000000001 + - type: recall_at_10 + value: 28.077999999999996 + - type: recall_at_100 + value: 51.403 + - type: recall_at_1000 + value: 72.406 + - type: recall_at_3 + value: 16.768 + - type: recall_at_5 + value: 20.737 + - task: + type: Retrieval + dataset: + type: dbpedia-entity + name: MTEB DBPedia + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 8.012 + - type: map_at_10 + value: 17.138 + - type: map_at_100 + value: 24.146 + - type: map_at_1000 + value: 25.622 + - type: map_at_3 + value: 12.552 + - type: map_at_5 + value: 14.435 + - type: mrr_at_1 + value: 62.25000000000001 + - type: mrr_at_10 + value: 71.186 + - type: mrr_at_100 + value: 71.504 + - type: mrr_at_1000 + value: 71.514 + - type: mrr_at_3 + value: 69.333 + - type: mrr_at_5 + value: 70.408 + - type: ndcg_at_1 + value: 49.75 + - type: ndcg_at_10 + value: 37.76 + - type: ndcg_at_100 + value: 42.071 + - type: ndcg_at_1000 + value: 49.309 + - type: ndcg_at_3 + value: 41.644 + - type: ndcg_at_5 + value: 39.812999999999995 + - type: precision_at_1 + value: 62.25000000000001 + - type: precision_at_10 + value: 30.15 + - type: precision_at_100 + value: 9.753 + - type: precision_at_1000 + value: 1.9189999999999998 + - type: precision_at_3 + value: 45.667 + - type: precision_at_5 + value: 39.15 + - type: recall_at_1 + value: 8.012 + - type: recall_at_10 + value: 22.599 + - type: recall_at_100 + value: 48.068 + - type: recall_at_1000 + value: 71.328 + - type: recall_at_3 + value: 14.043 + - type: recall_at_5 + value: 17.124 + - task: + type: Classification + dataset: + type: mteb/emotion + name: MTEB EmotionClassification + config: default + split: test + revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 + metrics: + - type: accuracy + value: 42.455 + - type: f1 + value: 37.59462649781862 + - task: + type: Retrieval + dataset: + type: fever + name: MTEB FEVER + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 58.092 + - type: map_at_10 + value: 69.586 + - type: map_at_100 + value: 69.968 + - type: map_at_1000 + value: 69.982 + - type: map_at_3 + value: 67.48100000000001 + - type: map_at_5 + value: 68.915 + - type: mrr_at_1 + value: 62.166 + - type: mrr_at_10 + value: 73.588 + - type: mrr_at_100 + value: 73.86399999999999 + - type: mrr_at_1000 + value: 73.868 + - type: mrr_at_3 + value: 71.6 + - type: mrr_at_5 + value: 72.99 + - type: ndcg_at_1 + value: 62.166 + - type: ndcg_at_10 + value: 75.27199999999999 + - type: ndcg_at_100 + value: 76.816 + - type: ndcg_at_1000 + value: 77.09700000000001 + - type: ndcg_at_3 + value: 71.36 + - type: ndcg_at_5 + value: 73.785 + - type: precision_at_1 + value: 62.166 + - type: precision_at_10 + value: 9.716 + - type: precision_at_100 + value: 1.065 + - type: precision_at_1000 + value: 0.11 + - type: precision_at_3 + value: 28.278 + - type: precision_at_5 + value: 18.343999999999998 + - type: recall_at_1 + value: 58.092 + - type: recall_at_10 + value: 88.73400000000001 + - type: recall_at_100 + value: 95.195 + - type: recall_at_1000 + value: 97.04599999999999 + - type: recall_at_3 + value: 78.45 + - type: recall_at_5 + value: 84.316 + - task: + type: Retrieval + dataset: + type: fiqa + name: MTEB FiQA2018 + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 16.649 + - type: map_at_10 + value: 26.457000000000004 + - type: map_at_100 + value: 28.169 + - type: map_at_1000 + value: 28.352 + - type: map_at_3 + value: 23.305 + - type: map_at_5 + value: 25.169000000000004 + - type: mrr_at_1 + value: 32.407000000000004 + - type: mrr_at_10 + value: 40.922 + - type: mrr_at_100 + value: 41.931000000000004 + - type: mrr_at_1000 + value: 41.983 + - type: mrr_at_3 + value: 38.786 + - type: mrr_at_5 + value: 40.205999999999996 + - type: ndcg_at_1 + value: 32.407000000000004 + - type: ndcg_at_10 + value: 33.314 + - type: ndcg_at_100 + value: 40.312 + - type: ndcg_at_1000 + value: 43.685 + - type: ndcg_at_3 + value: 30.391000000000002 + - type: ndcg_at_5 + value: 31.525 + - type: precision_at_1 + value: 32.407000000000004 + - type: precision_at_10 + value: 8.966000000000001 + - type: precision_at_100 + value: 1.6019999999999999 + - type: precision_at_1000 + value: 0.22200000000000003 + - type: precision_at_3 + value: 20.165 + - type: precision_at_5 + value: 14.722 + - type: recall_at_1 + value: 16.649 + - type: recall_at_10 + value: 39.117000000000004 + - type: recall_at_100 + value: 65.726 + - type: recall_at_1000 + value: 85.784 + - type: recall_at_3 + value: 27.914 + - type: recall_at_5 + value: 33.289 + - task: + type: Retrieval + dataset: + type: hotpotqa + name: MTEB HotpotQA + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 36.253 + - type: map_at_10 + value: 56.16799999999999 + - type: map_at_100 + value: 57.06099999999999 + - type: map_at_1000 + value: 57.126 + - type: map_at_3 + value: 52.644999999999996 + - type: map_at_5 + value: 54.909 + - type: mrr_at_1 + value: 72.505 + - type: mrr_at_10 + value: 79.66 + - type: mrr_at_100 + value: 79.869 + - type: mrr_at_1000 + value: 79.88 + - type: mrr_at_3 + value: 78.411 + - type: mrr_at_5 + value: 79.19800000000001 + - type: ndcg_at_1 + value: 72.505 + - type: ndcg_at_10 + value: 65.094 + - type: ndcg_at_100 + value: 68.219 + - type: ndcg_at_1000 + value: 69.515 + - type: ndcg_at_3 + value: 59.99 + - type: ndcg_at_5 + value: 62.909000000000006 + - type: precision_at_1 + value: 72.505 + - type: precision_at_10 + value: 13.749 + - type: precision_at_100 + value: 1.619 + - type: precision_at_1000 + value: 0.179 + - type: precision_at_3 + value: 38.357 + - type: precision_at_5 + value: 25.313000000000002 + - type: recall_at_1 + value: 36.253 + - type: recall_at_10 + value: 68.744 + - type: recall_at_100 + value: 80.925 + - type: recall_at_1000 + value: 89.534 + - type: recall_at_3 + value: 57.535000000000004 + - type: recall_at_5 + value: 63.282000000000004 + - task: + type: Classification + dataset: + type: mteb/imdb + name: MTEB ImdbClassification + config: default + split: test + revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 + metrics: + - type: accuracy + value: 80.82239999999999 + - type: ap + value: 75.65895781725314 + - type: f1 + value: 80.75880969095746 + - task: + type: Retrieval + dataset: + type: msmarco + name: MTEB MSMARCO + config: default + split: dev + revision: None + metrics: + - type: map_at_1 + value: 21.624 + - type: map_at_10 + value: 34.075 + - type: map_at_100 + value: 35.229 + - type: map_at_1000 + value: 35.276999999999994 + - type: map_at_3 + value: 30.245 + - type: map_at_5 + value: 32.42 + - type: mrr_at_1 + value: 22.264 + - type: mrr_at_10 + value: 34.638000000000005 + - type: mrr_at_100 + value: 35.744 + - type: mrr_at_1000 + value: 35.787 + - type: mrr_at_3 + value: 30.891000000000002 + - type: mrr_at_5 + value: 33.042 + - type: ndcg_at_1 + value: 22.264 + - type: ndcg_at_10 + value: 40.991 + - type: ndcg_at_100 + value: 46.563 + - type: ndcg_at_1000 + value: 47.743 + - type: ndcg_at_3 + value: 33.198 + - type: ndcg_at_5 + value: 37.069 + - type: precision_at_1 + value: 22.264 + - type: precision_at_10 + value: 6.5089999999999995 + - type: precision_at_100 + value: 0.9299999999999999 + - type: precision_at_1000 + value: 0.10300000000000001 + - type: precision_at_3 + value: 14.216999999999999 + - type: precision_at_5 + value: 10.487 + - type: recall_at_1 + value: 21.624 + - type: recall_at_10 + value: 62.303 + - type: recall_at_100 + value: 88.124 + - type: recall_at_1000 + value: 97.08 + - type: recall_at_3 + value: 41.099999999999994 + - type: recall_at_5 + value: 50.381 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (en) + config: en + split: test + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + metrics: + - type: accuracy + value: 91.06703146374831 + - type: f1 + value: 90.86867815863172 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (de) + config: de + split: test + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + metrics: + - type: accuracy + value: 87.46970977740209 + - type: f1 + value: 86.36832872036588 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (es) + config: es + split: test + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + metrics: + - type: accuracy + value: 89.26951300867245 + - type: f1 + value: 88.93561193959502 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (fr) + config: fr + split: test + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + metrics: + - type: accuracy + value: 84.22799874725963 + - type: f1 + value: 84.30490069236556 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (hi) + config: hi + split: test + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + metrics: + - type: accuracy + value: 86.02007888131948 + - type: f1 + value: 85.39376041027991 + - task: + type: Classification + dataset: + type: mteb/mtop_domain + name: MTEB MTOPDomainClassification (th) + config: th + split: test + revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf + metrics: + - type: accuracy + value: 85.34900542495481 + - type: f1 + value: 85.39859673336713 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (en) + config: en + split: test + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + metrics: + - type: accuracy + value: 71.078431372549 + - type: f1 + value: 53.45071102002276 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (de) + config: de + split: test + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + metrics: + - type: accuracy + value: 65.85798816568047 + - type: f1 + value: 46.53112748993529 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (es) + config: es + split: test + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + metrics: + - type: accuracy + value: 67.96864576384256 + - type: f1 + value: 45.966703022829506 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (fr) + config: fr + split: test + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + metrics: + - type: accuracy + value: 61.31537738803633 + - type: f1 + value: 45.52601712835461 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (hi) + config: hi + split: test + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + metrics: + - type: accuracy + value: 66.29616349946218 + - type: f1 + value: 47.24166485726613 + - task: + type: Classification + dataset: + type: mteb/mtop_intent + name: MTEB MTOPIntentClassification (th) + config: th + split: test + revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba + metrics: + - type: accuracy + value: 67.51537070524412 + - type: f1 + value: 49.463476319014276 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (af) + config: af + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 57.06792199058508 + - type: f1 + value: 54.094921857502285 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (am) + config: am + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 51.960322797579025 + - type: f1 + value: 48.547371223370945 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ar) + config: ar + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 54.425016812373904 + - type: f1 + value: 50.47069202054312 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (az) + config: az + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 59.798251513113655 + - type: f1 + value: 57.05013069086648 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (bn) + config: bn + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 59.37794216543376 + - type: f1 + value: 56.3607992649805 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (cy) + config: cy + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 46.56018829858777 + - type: f1 + value: 43.87319715715134 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (da) + config: da + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 62.9724277067922 + - type: f1 + value: 59.36480066245562 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (de) + config: de + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 62.72696704774715 + - type: f1 + value: 59.143595966615855 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (el) + config: el + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 61.5971755211836 + - type: f1 + value: 59.169445724946726 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (en) + config: en + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 70.29589778076665 + - type: f1 + value: 67.7577001808977 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (es) + config: es + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 66.31136516476126 + - type: f1 + value: 64.52032955983242 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (fa) + config: fa + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 65.54472091459314 + - type: f1 + value: 61.47903120066317 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (fi) + config: fi + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 61.45595158036314 + - type: f1 + value: 58.0891846024637 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (fr) + config: fr + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 65.47074646940149 + - type: f1 + value: 62.84830858877575 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (he) + config: he + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 58.046402151983855 + - type: f1 + value: 55.269074430533195 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (hi) + config: hi + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 64.06523201075991 + - type: f1 + value: 61.35339643021369 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (hu) + config: hu + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 60.954942837928726 + - type: f1 + value: 57.07035922704846 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (hy) + config: hy + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 57.404169468728995 + - type: f1 + value: 53.94259011839138 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (id) + config: id + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 64.16610625420309 + - type: f1 + value: 61.337103431499365 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (is) + config: is + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 52.262945527908535 + - type: f1 + value: 49.7610691598921 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (it) + config: it + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 65.54472091459314 + - type: f1 + value: 63.469099018440154 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ja) + config: ja + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 68.22797579018157 + - type: f1 + value: 64.89098471083001 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (jv) + config: jv + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 50.847343644922674 + - type: f1 + value: 47.8536963168393 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ka) + config: ka + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 48.45326160053799 + - type: f1 + value: 46.370078045805556 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (km) + config: km + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 42.83120376597175 + - type: f1 + value: 39.68948521599982 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (kn) + config: kn + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 57.5084061869536 + - type: f1 + value: 53.961876160401545 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ko) + config: ko + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 63.7895090786819 + - type: f1 + value: 61.134223684676 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (lv) + config: lv + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 54.98991257565569 + - type: f1 + value: 52.579862862826296 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ml) + config: ml + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 61.90316072629456 + - type: f1 + value: 58.203024538290336 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (mn) + config: mn + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 57.09818426361802 + - type: f1 + value: 54.22718458445455 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ms) + config: ms + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 58.991257565568255 + - type: f1 + value: 55.84892781767421 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (my) + config: my + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 55.901143241425686 + - type: f1 + value: 52.25264332199797 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (nb) + config: nb + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 61.96368527236047 + - type: f1 + value: 58.927243876153454 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (nl) + config: nl + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 65.64223268325489 + - type: f1 + value: 62.340453718379706 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (pl) + config: pl + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 64.52589105581708 + - type: f1 + value: 61.661113187022174 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (pt) + config: pt + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 66.84599865501009 + - type: f1 + value: 64.59342572873005 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ro) + config: ro + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 60.81035642232684 + - type: f1 + value: 57.5169089806797 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ru) + config: ru + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 65.75991930060525 + - type: f1 + value: 62.89531115787938 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sl) + config: sl + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 56.51647612642906 + - type: f1 + value: 54.33154780100043 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sq) + config: sq + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 57.985877605917956 + - type: f1 + value: 54.46187524463802 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sv) + config: sv + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 65.03026227303296 + - type: f1 + value: 62.34377392877748 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (sw) + config: sw + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 53.567585743106925 + - type: f1 + value: 50.73770655983206 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ta) + config: ta + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 57.2595830531271 + - type: f1 + value: 53.657327291708626 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (te) + config: te + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 57.82784129119032 + - type: f1 + value: 54.82518072665301 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (th) + config: th + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 64.06859448554137 + - type: f1 + value: 63.00185280500495 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (tl) + config: tl + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 58.91055817081371 + - type: f1 + value: 55.54116301224262 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (tr) + config: tr + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 63.54404841963686 + - type: f1 + value: 59.57650946030184 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (ur) + config: ur + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 59.27706792199059 + - type: f1 + value: 56.50010066083435 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (vi) + config: vi + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 64.0719569603228 + - type: f1 + value: 61.817075925647956 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (zh-CN) + config: zh-CN + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 68.23806321452591 + - type: f1 + value: 65.24917026029749 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_intent + name: MTEB MassiveIntentClassification (zh-TW) + config: zh-TW + split: test + revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 + metrics: + - type: accuracy + value: 62.53530598520511 + - type: f1 + value: 61.71131132295768 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (af) + config: af + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 63.04303967720243 + - type: f1 + value: 60.3950085685985 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (am) + config: am + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 56.83591123066578 + - type: f1 + value: 54.95059828830849 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ar) + config: ar + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 59.62340282447881 + - type: f1 + value: 59.525159996498225 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (az) + config: az + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 60.85406859448555 + - type: f1 + value: 59.129299095681276 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (bn) + config: bn + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 62.76731674512441 + - type: f1 + value: 61.159560612627715 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (cy) + config: cy + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 50.181573638197705 + - type: f1 + value: 46.98422176289957 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (da) + config: da + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 68.92737054472092 + - type: f1 + value: 67.69135611952979 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (de) + config: de + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 69.18964357767318 + - type: f1 + value: 68.46106138186214 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (el) + config: el + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 67.0712844653665 + - type: f1 + value: 66.75545422473901 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (en) + config: en + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 74.4754539340955 + - type: f1 + value: 74.38427146553252 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (es) + config: es + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 69.82515131136518 + - type: f1 + value: 69.63516462173847 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (fa) + config: fa + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 68.70880968392737 + - type: f1 + value: 67.45420662567926 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (fi) + config: fi + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 65.95494283792871 + - type: f1 + value: 65.06191009049222 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (fr) + config: fr + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 68.75924680564896 + - type: f1 + value: 68.30833379585945 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (he) + config: he + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 63.806321452589096 + - type: f1 + value: 63.273048243765054 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (hi) + config: hi + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 67.68997982515133 + - type: f1 + value: 66.54703855381324 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (hu) + config: hu + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 66.46940147948891 + - type: f1 + value: 65.91017343463396 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (hy) + config: hy + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 59.49899125756556 + - type: f1 + value: 57.90333469917769 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (id) + config: id + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 67.9219905850706 + - type: f1 + value: 67.23169403762938 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (is) + config: is + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 56.486213853396094 + - type: f1 + value: 54.85282355583758 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (it) + config: it + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 69.04169468728985 + - type: f1 + value: 68.83833333320462 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ja) + config: ja + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 73.88702084734365 + - type: f1 + value: 74.04474735232299 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (jv) + config: jv + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 56.63416274377943 + - type: f1 + value: 55.11332211687954 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ka) + config: ka + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 52.23604572965702 + - type: f1 + value: 50.86529813991055 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (km) + config: km + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 46.62407531943511 + - type: f1 + value: 43.63485467164535 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (kn) + config: kn + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 59.15601882985878 + - type: f1 + value: 57.522837510959924 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ko) + config: ko + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 69.84532616005382 + - type: f1 + value: 69.60021127179697 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (lv) + config: lv + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 56.65770006724949 + - type: f1 + value: 55.84219135523227 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ml) + config: ml + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 66.53665097511768 + - type: f1 + value: 65.09087787792639 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (mn) + config: mn + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 59.31405514458642 + - type: f1 + value: 58.06135303831491 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ms) + config: ms + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 64.88231338264964 + - type: f1 + value: 62.751099407787926 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (my) + config: my + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 58.86012104909213 + - type: f1 + value: 56.29118323058282 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (nb) + config: nb + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 67.37390719569602 + - type: f1 + value: 66.27922244885102 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (nl) + config: nl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 70.8675184936113 + - type: f1 + value: 70.22146529932019 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (pl) + config: pl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 68.2212508406187 + - type: f1 + value: 67.77454802056282 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (pt) + config: pt + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 68.18090114324143 + - type: f1 + value: 68.03737625431621 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ro) + config: ro + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 64.65030262273034 + - type: f1 + value: 63.792945486912856 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ru) + config: ru + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 69.48217888365838 + - type: f1 + value: 69.96028997292197 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sl) + config: sl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 60.17821116341627 + - type: f1 + value: 59.3935969827171 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sq) + config: sq + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 62.86146603900471 + - type: f1 + value: 60.133692735032376 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sv) + config: sv + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 70.89441829186282 + - type: f1 + value: 70.03064076194089 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (sw) + config: sw + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 58.15063887020847 + - type: f1 + value: 56.23326278499678 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ta) + config: ta + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 59.43846671149966 + - type: f1 + value: 57.70440450281974 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (te) + config: te + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 60.8507061197041 + - type: f1 + value: 59.22916396061171 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (th) + config: th + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 70.65568258238063 + - type: f1 + value: 69.90736239440633 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (tl) + config: tl + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 60.8843308675185 + - type: f1 + value: 59.30332663713599 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (tr) + config: tr + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 68.05312710154674 + - type: f1 + value: 67.44024062594775 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (ur) + config: ur + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 62.111634162743776 + - type: f1 + value: 60.89083013084519 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (vi) + config: vi + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 67.44115669132482 + - type: f1 + value: 67.92227541674552 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (zh-CN) + config: zh-CN + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 74.4687289845326 + - type: f1 + value: 74.16376793486025 + - task: + type: Classification + dataset: + type: mteb/amazon_massive_scenario + name: MTEB MassiveScenarioClassification (zh-TW) + config: zh-TW + split: test + revision: 7d571f92784cd94a019292a1f45445077d0ef634 + metrics: + - type: accuracy + value: 68.31876260928043 + - type: f1 + value: 68.5246745215607 + - task: + type: Clustering + dataset: + type: mteb/medrxiv-clustering-p2p + name: MTEB MedrxivClusteringP2P + config: default + split: test + revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 + metrics: + - type: v_measure + value: 30.90431696479766 + - task: + type: Clustering + dataset: + type: mteb/medrxiv-clustering-s2s + name: MTEB MedrxivClusteringS2S + config: default + split: test + revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 + metrics: + - type: v_measure + value: 27.259158476693774 + - task: + type: Reranking + dataset: + type: mteb/mind_small + name: MTEB MindSmallReranking + config: default + split: test + revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 + metrics: + - type: map + value: 30.28445330838555 + - type: mrr + value: 31.15758529581164 + - task: + type: Retrieval + dataset: + type: nfcorpus + name: MTEB NFCorpus + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 5.353 + - type: map_at_10 + value: 11.565 + - type: map_at_100 + value: 14.097000000000001 + - type: map_at_1000 + value: 15.354999999999999 + - type: map_at_3 + value: 8.749 + - type: map_at_5 + value: 9.974 + - type: mrr_at_1 + value: 42.105 + - type: mrr_at_10 + value: 50.589 + - type: mrr_at_100 + value: 51.187000000000005 + - type: mrr_at_1000 + value: 51.233 + - type: mrr_at_3 + value: 48.246 + - type: mrr_at_5 + value: 49.546 + - type: ndcg_at_1 + value: 40.402 + - type: ndcg_at_10 + value: 31.009999999999998 + - type: ndcg_at_100 + value: 28.026 + - type: ndcg_at_1000 + value: 36.905 + - type: ndcg_at_3 + value: 35.983 + - type: ndcg_at_5 + value: 33.764 + - type: precision_at_1 + value: 42.105 + - type: precision_at_10 + value: 22.786 + - type: precision_at_100 + value: 6.916 + - type: precision_at_1000 + value: 1.981 + - type: precision_at_3 + value: 33.333 + - type: precision_at_5 + value: 28.731 + - type: recall_at_1 + value: 5.353 + - type: recall_at_10 + value: 15.039 + - type: recall_at_100 + value: 27.348 + - type: recall_at_1000 + value: 59.453 + - type: recall_at_3 + value: 9.792 + - type: recall_at_5 + value: 11.882 + - task: + type: Retrieval + dataset: + type: nq + name: MTEB NQ + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 33.852 + - type: map_at_10 + value: 48.924 + - type: map_at_100 + value: 49.854 + - type: map_at_1000 + value: 49.886 + - type: map_at_3 + value: 44.9 + - type: map_at_5 + value: 47.387 + - type: mrr_at_1 + value: 38.035999999999994 + - type: mrr_at_10 + value: 51.644 + - type: mrr_at_100 + value: 52.339 + - type: mrr_at_1000 + value: 52.35999999999999 + - type: mrr_at_3 + value: 48.421 + - type: mrr_at_5 + value: 50.468999999999994 + - type: ndcg_at_1 + value: 38.007000000000005 + - type: ndcg_at_10 + value: 56.293000000000006 + - type: ndcg_at_100 + value: 60.167 + - type: ndcg_at_1000 + value: 60.916000000000004 + - type: ndcg_at_3 + value: 48.903999999999996 + - type: ndcg_at_5 + value: 52.978 + - type: precision_at_1 + value: 38.007000000000005 + - type: precision_at_10 + value: 9.041 + - type: precision_at_100 + value: 1.1199999999999999 + - type: precision_at_1000 + value: 0.11900000000000001 + - type: precision_at_3 + value: 22.084 + - type: precision_at_5 + value: 15.608 + - type: recall_at_1 + value: 33.852 + - type: recall_at_10 + value: 75.893 + - type: recall_at_100 + value: 92.589 + - type: recall_at_1000 + value: 98.153 + - type: recall_at_3 + value: 56.969 + - type: recall_at_5 + value: 66.283 + - task: + type: Retrieval + dataset: + type: quora + name: MTEB QuoraRetrieval + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 69.174 + - type: map_at_10 + value: 82.891 + - type: map_at_100 + value: 83.545 + - type: map_at_1000 + value: 83.56700000000001 + - type: map_at_3 + value: 79.944 + - type: map_at_5 + value: 81.812 + - type: mrr_at_1 + value: 79.67999999999999 + - type: mrr_at_10 + value: 86.279 + - type: mrr_at_100 + value: 86.39 + - type: mrr_at_1000 + value: 86.392 + - type: mrr_at_3 + value: 85.21 + - type: mrr_at_5 + value: 85.92999999999999 + - type: ndcg_at_1 + value: 79.69000000000001 + - type: ndcg_at_10 + value: 86.929 + - type: ndcg_at_100 + value: 88.266 + - type: ndcg_at_1000 + value: 88.428 + - type: ndcg_at_3 + value: 83.899 + - type: ndcg_at_5 + value: 85.56700000000001 + - type: precision_at_1 + value: 79.69000000000001 + - type: precision_at_10 + value: 13.161000000000001 + - type: precision_at_100 + value: 1.513 + - type: precision_at_1000 + value: 0.156 + - type: precision_at_3 + value: 36.603 + - type: precision_at_5 + value: 24.138 + - type: recall_at_1 + value: 69.174 + - type: recall_at_10 + value: 94.529 + - type: recall_at_100 + value: 99.15 + - type: recall_at_1000 + value: 99.925 + - type: recall_at_3 + value: 85.86200000000001 + - type: recall_at_5 + value: 90.501 + - task: + type: Clustering + dataset: + type: mteb/reddit-clustering + name: MTEB RedditClustering + config: default + split: test + revision: 24640382cdbf8abc73003fb0fa6d111a705499eb + metrics: + - type: v_measure + value: 39.13064340585255 + - task: + type: Clustering + dataset: + type: mteb/reddit-clustering-p2p + name: MTEB RedditClusteringP2P + config: default + split: test + revision: 282350215ef01743dc01b456c7f5241fa8937f16 + metrics: + - type: v_measure + value: 58.97884249325877 + - task: + type: Retrieval + dataset: + type: scidocs + name: MTEB SCIDOCS + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 3.4680000000000004 + - type: map_at_10 + value: 7.865 + - type: map_at_100 + value: 9.332 + - type: map_at_1000 + value: 9.587 + - type: map_at_3 + value: 5.800000000000001 + - type: map_at_5 + value: 6.8790000000000004 + - type: mrr_at_1 + value: 17.0 + - type: mrr_at_10 + value: 25.629 + - type: mrr_at_100 + value: 26.806 + - type: mrr_at_1000 + value: 26.889000000000003 + - type: mrr_at_3 + value: 22.8 + - type: mrr_at_5 + value: 24.26 + - type: ndcg_at_1 + value: 17.0 + - type: ndcg_at_10 + value: 13.895 + - type: ndcg_at_100 + value: 20.491999999999997 + - type: ndcg_at_1000 + value: 25.759999999999998 + - type: ndcg_at_3 + value: 13.347999999999999 + - type: ndcg_at_5 + value: 11.61 + - type: precision_at_1 + value: 17.0 + - type: precision_at_10 + value: 7.090000000000001 + - type: precision_at_100 + value: 1.669 + - type: precision_at_1000 + value: 0.294 + - type: precision_at_3 + value: 12.3 + - type: precision_at_5 + value: 10.02 + - type: recall_at_1 + value: 3.4680000000000004 + - type: recall_at_10 + value: 14.363000000000001 + - type: recall_at_100 + value: 33.875 + - type: recall_at_1000 + value: 59.711999999999996 + - type: recall_at_3 + value: 7.483 + - type: recall_at_5 + value: 10.173 + - task: + type: STS + dataset: + type: mteb/sickr-sts + name: MTEB SICK-R + config: default + split: test + revision: a6ea5a8cab320b040a23452cc28066d9beae2cee + metrics: + - type: cos_sim_pearson + value: 83.04084311714061 + - type: cos_sim_spearman + value: 77.51342467443078 + - type: euclidean_pearson + value: 80.0321166028479 + - type: euclidean_spearman + value: 77.29249114733226 + - type: manhattan_pearson + value: 80.03105964262431 + - type: manhattan_spearman + value: 77.22373689514794 + - task: + type: STS + dataset: + type: mteb/sts12-sts + name: MTEB STS12 + config: default + split: test + revision: a0d554a64d88156834ff5ae9920b964011b16384 + metrics: + - type: cos_sim_pearson + value: 84.1680158034387 + - type: cos_sim_spearman + value: 76.55983344071117 + - type: euclidean_pearson + value: 79.75266678300143 + - type: euclidean_spearman + value: 75.34516823467025 + - type: manhattan_pearson + value: 79.75959151517357 + - type: manhattan_spearman + value: 75.42330344141912 + - task: + type: STS + dataset: + type: mteb/sts13-sts + name: MTEB STS13 + config: default + split: test + revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca + metrics: + - type: cos_sim_pearson + value: 76.48898993209346 + - type: cos_sim_spearman + value: 76.96954120323366 + - type: euclidean_pearson + value: 76.94139109279668 + - type: euclidean_spearman + value: 76.85860283201711 + - type: manhattan_pearson + value: 76.6944095091912 + - type: manhattan_spearman + value: 76.61096912972553 + - task: + type: STS + dataset: + type: mteb/sts14-sts + name: MTEB STS14 + config: default + split: test + revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 + metrics: + - type: cos_sim_pearson + value: 77.85082366246944 + - type: cos_sim_spearman + value: 75.52053350101731 + - type: euclidean_pearson + value: 77.1165845070926 + - type: euclidean_spearman + value: 75.31216065884388 + - type: manhattan_pearson + value: 77.06193941833494 + - type: manhattan_spearman + value: 75.31003701700112 + - task: + type: STS + dataset: + type: mteb/sts15-sts + name: MTEB STS15 + config: default + split: test + revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 + metrics: + - type: cos_sim_pearson + value: 86.36305246526497 + - type: cos_sim_spearman + value: 87.11704613927415 + - type: euclidean_pearson + value: 86.04199125810939 + - type: euclidean_spearman + value: 86.51117572414263 + - type: manhattan_pearson + value: 86.0805106816633 + - type: manhattan_spearman + value: 86.52798366512229 + - task: + type: STS + dataset: + type: mteb/sts16-sts + name: MTEB STS16 + config: default + split: test + revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 + metrics: + - type: cos_sim_pearson + value: 82.18536255599724 + - type: cos_sim_spearman + value: 83.63377151025418 + - type: euclidean_pearson + value: 83.24657467993141 + - type: euclidean_spearman + value: 84.02751481993825 + - type: manhattan_pearson + value: 83.11941806582371 + - type: manhattan_spearman + value: 83.84251281019304 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (ko-ko) + config: ko-ko + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 78.95816528475514 + - type: cos_sim_spearman + value: 78.86607380120462 + - type: euclidean_pearson + value: 78.51268699230545 + - type: euclidean_spearman + value: 79.11649316502229 + - type: manhattan_pearson + value: 78.32367302808157 + - type: manhattan_spearman + value: 78.90277699624637 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (ar-ar) + config: ar-ar + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 72.89126914997624 + - type: cos_sim_spearman + value: 73.0296921832678 + - type: euclidean_pearson + value: 71.50385903677738 + - type: euclidean_spearman + value: 73.13368899716289 + - type: manhattan_pearson + value: 71.47421463379519 + - type: manhattan_spearman + value: 73.03383242946575 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-ar) + config: en-ar + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 59.22923684492637 + - type: cos_sim_spearman + value: 57.41013211368396 + - type: euclidean_pearson + value: 61.21107388080905 + - type: euclidean_spearman + value: 60.07620768697254 + - type: manhattan_pearson + value: 59.60157142786555 + - type: manhattan_spearman + value: 59.14069604103739 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-de) + config: en-de + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 76.24345978774299 + - type: cos_sim_spearman + value: 77.24225743830719 + - type: euclidean_pearson + value: 76.66226095469165 + - type: euclidean_spearman + value: 77.60708820493146 + - type: manhattan_pearson + value: 76.05303324760429 + - type: manhattan_spearman + value: 76.96353149912348 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-en) + config: en-en + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 85.50879160160852 + - type: cos_sim_spearman + value: 86.43594662965224 + - type: euclidean_pearson + value: 86.06846012826577 + - type: euclidean_spearman + value: 86.02041395794136 + - type: manhattan_pearson + value: 86.10916255616904 + - type: manhattan_spearman + value: 86.07346068198953 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (en-tr) + config: en-tr + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 58.39803698977196 + - type: cos_sim_spearman + value: 55.96910950423142 + - type: euclidean_pearson + value: 58.17941175613059 + - type: euclidean_spearman + value: 55.03019330522745 + - type: manhattan_pearson + value: 57.333358138183286 + - type: manhattan_spearman + value: 54.04614023149965 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (es-en) + config: es-en + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 70.98304089637197 + - type: cos_sim_spearman + value: 72.44071656215888 + - type: euclidean_pearson + value: 72.19224359033983 + - type: euclidean_spearman + value: 73.89871188913025 + - type: manhattan_pearson + value: 71.21098311547406 + - type: manhattan_spearman + value: 72.93405764824821 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (es-es) + config: es-es + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 85.99792397466308 + - type: cos_sim_spearman + value: 84.83824377879495 + - type: euclidean_pearson + value: 85.70043288694438 + - type: euclidean_spearman + value: 84.70627558703686 + - type: manhattan_pearson + value: 85.89570850150801 + - type: manhattan_spearman + value: 84.95806105313007 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (fr-en) + config: fr-en + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 72.21850322994712 + - type: cos_sim_spearman + value: 72.28669398117248 + - type: euclidean_pearson + value: 73.40082510412948 + - type: euclidean_spearman + value: 73.0326539281865 + - type: manhattan_pearson + value: 71.8659633964841 + - type: manhattan_spearman + value: 71.57817425823303 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (it-en) + config: it-en + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 75.80921368595645 + - type: cos_sim_spearman + value: 77.33209091229315 + - type: euclidean_pearson + value: 76.53159540154829 + - type: euclidean_spearman + value: 78.17960842810093 + - type: manhattan_pearson + value: 76.13530186637601 + - type: manhattan_spearman + value: 78.00701437666875 + - task: + type: STS + dataset: + type: mteb/sts17-crosslingual-sts + name: MTEB STS17 (nl-en) + config: nl-en + split: test + revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d + metrics: + - type: cos_sim_pearson + value: 74.74980608267349 + - type: cos_sim_spearman + value: 75.37597374318821 + - type: euclidean_pearson + value: 74.90506081911661 + - type: euclidean_spearman + value: 75.30151613124521 + - type: manhattan_pearson + value: 74.62642745918002 + - type: manhattan_spearman + value: 75.18619716592303 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (en) + config: en + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 59.632662289205584 + - type: cos_sim_spearman + value: 60.938543391610914 + - type: euclidean_pearson + value: 62.113200529767056 + - type: euclidean_spearman + value: 61.410312633261164 + - type: manhattan_pearson + value: 61.75494698945686 + - type: manhattan_spearman + value: 60.92726195322362 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de) + config: de + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 45.283470551557244 + - type: cos_sim_spearman + value: 53.44833015864201 + - type: euclidean_pearson + value: 41.17892011120893 + - type: euclidean_spearman + value: 53.81441383126767 + - type: manhattan_pearson + value: 41.17482200420659 + - type: manhattan_spearman + value: 53.82180269276363 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (es) + config: es + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 60.5069165306236 + - type: cos_sim_spearman + value: 66.87803259033826 + - type: euclidean_pearson + value: 63.5428979418236 + - type: euclidean_spearman + value: 66.9293576586897 + - type: manhattan_pearson + value: 63.59789526178922 + - type: manhattan_spearman + value: 66.86555009875066 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (pl) + config: pl + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 28.23026196280264 + - type: cos_sim_spearman + value: 35.79397812652861 + - type: euclidean_pearson + value: 17.828102102767353 + - type: euclidean_spearman + value: 35.721501145568894 + - type: manhattan_pearson + value: 17.77134274219677 + - type: manhattan_spearman + value: 35.98107902846267 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (tr) + config: tr + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 56.51946541393812 + - type: cos_sim_spearman + value: 63.714686006214485 + - type: euclidean_pearson + value: 58.32104651305898 + - type: euclidean_spearman + value: 62.237110895702216 + - type: manhattan_pearson + value: 58.579416468759185 + - type: manhattan_spearman + value: 62.459738981727 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (ar) + config: ar + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 48.76009839569795 + - type: cos_sim_spearman + value: 56.65188431953149 + - type: euclidean_pearson + value: 50.997682160915595 + - type: euclidean_spearman + value: 55.99910008818135 + - type: manhattan_pearson + value: 50.76220659606342 + - type: manhattan_spearman + value: 55.517347595391456 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (ru) + config: ru + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 51.232731157702425 + - type: cos_sim_spearman + value: 59.89531877658345 + - type: euclidean_pearson + value: 49.937914570348376 + - type: euclidean_spearman + value: 60.220905659334036 + - type: manhattan_pearson + value: 50.00987996844193 + - type: manhattan_spearman + value: 60.081341480977926 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (zh) + config: zh + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 54.717524559088005 + - type: cos_sim_spearman + value: 66.83570886252286 + - type: euclidean_pearson + value: 58.41338625505467 + - type: euclidean_spearman + value: 66.68991427704938 + - type: manhattan_pearson + value: 58.78638572916807 + - type: manhattan_spearman + value: 66.58684161046335 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (fr) + config: fr + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 73.2962042954962 + - type: cos_sim_spearman + value: 76.58255504852025 + - type: euclidean_pearson + value: 75.70983192778257 + - type: euclidean_spearman + value: 77.4547684870542 + - type: manhattan_pearson + value: 75.75565853870485 + - type: manhattan_spearman + value: 76.90208974949428 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de-en) + config: de-en + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 54.47396266924846 + - type: cos_sim_spearman + value: 56.492267162048606 + - type: euclidean_pearson + value: 55.998505203070195 + - type: euclidean_spearman + value: 56.46447012960222 + - type: manhattan_pearson + value: 54.873172394430995 + - type: manhattan_spearman + value: 56.58111534551218 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (es-en) + config: es-en + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 69.87177267688686 + - type: cos_sim_spearman + value: 74.57160943395763 + - type: euclidean_pearson + value: 70.88330406826788 + - type: euclidean_spearman + value: 74.29767636038422 + - type: manhattan_pearson + value: 71.38245248369536 + - type: manhattan_spearman + value: 74.53102232732175 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (it) + config: it + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 72.80225656959544 + - type: cos_sim_spearman + value: 76.52646173725735 + - type: euclidean_pearson + value: 73.95710720200799 + - type: euclidean_spearman + value: 76.54040031984111 + - type: manhattan_pearson + value: 73.89679971946774 + - type: manhattan_spearman + value: 76.60886958161574 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (pl-en) + config: pl-en + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 70.70844249898789 + - type: cos_sim_spearman + value: 72.68571783670241 + - type: euclidean_pearson + value: 72.38800772441031 + - type: euclidean_spearman + value: 72.86804422703312 + - type: manhattan_pearson + value: 71.29840508203515 + - type: manhattan_spearman + value: 71.86264441749513 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (zh-en) + config: zh-en + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 58.647478923935694 + - type: cos_sim_spearman + value: 63.74453623540931 + - type: euclidean_pearson + value: 59.60138032437505 + - type: euclidean_spearman + value: 63.947930832166065 + - type: manhattan_pearson + value: 58.59735509491861 + - type: manhattan_spearman + value: 62.082503844627404 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (es-it) + config: es-it + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 65.8722516867162 + - type: cos_sim_spearman + value: 71.81208592523012 + - type: euclidean_pearson + value: 67.95315252165956 + - type: euclidean_spearman + value: 73.00749822046009 + - type: manhattan_pearson + value: 68.07884688638924 + - type: manhattan_spearman + value: 72.34210325803069 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de-fr) + config: de-fr + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 54.5405814240949 + - type: cos_sim_spearman + value: 60.56838649023775 + - type: euclidean_pearson + value: 53.011731611314104 + - type: euclidean_spearman + value: 58.533194841668426 + - type: manhattan_pearson + value: 53.623067729338494 + - type: manhattan_spearman + value: 58.018756154446926 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (de-pl) + config: de-pl + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 13.611046866216112 + - type: cos_sim_spearman + value: 28.238192909158492 + - type: euclidean_pearson + value: 22.16189199885129 + - type: euclidean_spearman + value: 35.012895679076564 + - type: manhattan_pearson + value: 21.969771178698387 + - type: manhattan_spearman + value: 32.456985088607475 + - task: + type: STS + dataset: + type: mteb/sts22-crosslingual-sts + name: MTEB STS22 (fr-pl) + config: fr-pl + split: test + revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 + metrics: + - type: cos_sim_pearson + value: 74.58077407011655 + - type: cos_sim_spearman + value: 84.51542547285167 + - type: euclidean_pearson + value: 74.64613843596234 + - type: euclidean_spearman + value: 84.51542547285167 + - type: manhattan_pearson + value: 75.15335973101396 + - type: manhattan_spearman + value: 84.51542547285167 + - task: + type: STS + dataset: + type: mteb/stsbenchmark-sts + name: MTEB STSBenchmark + config: default + split: test + revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 + metrics: + - type: cos_sim_pearson + value: 82.0739825531578 + - type: cos_sim_spearman + value: 84.01057479311115 + - type: euclidean_pearson + value: 83.85453227433344 + - type: euclidean_spearman + value: 84.01630226898655 + - type: manhattan_pearson + value: 83.75323603028978 + - type: manhattan_spearman + value: 83.89677983727685 + - task: + type: Reranking + dataset: + type: mteb/scidocs-reranking + name: MTEB SciDocsRR + config: default + split: test + revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab + metrics: + - type: map + value: 78.12945623123957 + - type: mrr + value: 93.87738713719106 + - task: + type: Retrieval + dataset: + type: scifact + name: MTEB SciFact + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 52.983000000000004 + - type: map_at_10 + value: 62.946000000000005 + - type: map_at_100 + value: 63.514 + - type: map_at_1000 + value: 63.554 + - type: map_at_3 + value: 60.183 + - type: map_at_5 + value: 61.672000000000004 + - type: mrr_at_1 + value: 55.667 + - type: mrr_at_10 + value: 64.522 + - type: mrr_at_100 + value: 64.957 + - type: mrr_at_1000 + value: 64.995 + - type: mrr_at_3 + value: 62.388999999999996 + - type: mrr_at_5 + value: 63.639 + - type: ndcg_at_1 + value: 55.667 + - type: ndcg_at_10 + value: 67.704 + - type: ndcg_at_100 + value: 70.299 + - type: ndcg_at_1000 + value: 71.241 + - type: ndcg_at_3 + value: 62.866 + - type: ndcg_at_5 + value: 65.16999999999999 + - type: precision_at_1 + value: 55.667 + - type: precision_at_10 + value: 9.033 + - type: precision_at_100 + value: 1.053 + - type: precision_at_1000 + value: 0.11299999999999999 + - type: precision_at_3 + value: 24.444 + - type: precision_at_5 + value: 16.133 + - type: recall_at_1 + value: 52.983000000000004 + - type: recall_at_10 + value: 80.656 + - type: recall_at_100 + value: 92.5 + - type: recall_at_1000 + value: 99.667 + - type: recall_at_3 + value: 67.744 + - type: recall_at_5 + value: 73.433 + - task: + type: PairClassification + dataset: + type: mteb/sprintduplicatequestions-pairclassification + name: MTEB SprintDuplicateQuestions + config: default + split: test + revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 + metrics: + - type: cos_sim_accuracy + value: 99.72772277227723 + - type: cos_sim_ap + value: 92.17845897992215 + - type: cos_sim_f1 + value: 85.9746835443038 + - type: cos_sim_precision + value: 87.07692307692308 + - type: cos_sim_recall + value: 84.89999999999999 + - type: dot_accuracy + value: 99.3039603960396 + - type: dot_ap + value: 60.70244020124878 + - type: dot_f1 + value: 59.92742353551063 + - type: dot_precision + value: 62.21743810548978 + - type: dot_recall + value: 57.8 + - type: euclidean_accuracy + value: 99.71683168316832 + - type: euclidean_ap + value: 91.53997039964659 + - type: euclidean_f1 + value: 84.88372093023257 + - type: euclidean_precision + value: 90.02242152466367 + - type: euclidean_recall + value: 80.30000000000001 + - type: manhattan_accuracy + value: 99.72376237623763 + - type: manhattan_ap + value: 91.80756777790289 + - type: manhattan_f1 + value: 85.48468106479157 + - type: manhattan_precision + value: 85.8728557013118 + - type: manhattan_recall + value: 85.1 + - type: max_accuracy + value: 99.72772277227723 + - type: max_ap + value: 92.17845897992215 + - type: max_f1 + value: 85.9746835443038 + - task: + type: Clustering + dataset: + type: mteb/stackexchange-clustering + name: MTEB StackExchangeClustering + config: default + split: test + revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 + metrics: + - type: v_measure + value: 53.52464042600003 + - task: + type: Clustering + dataset: + type: mteb/stackexchange-clustering-p2p + name: MTEB StackExchangeClusteringP2P + config: default + split: test + revision: 815ca46b2622cec33ccafc3735d572c266efdb44 + metrics: + - type: v_measure + value: 32.071631948736 + - task: + type: Reranking + dataset: + type: mteb/stackoverflowdupquestions-reranking + name: MTEB StackOverflowDupQuestions + config: default + split: test + revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 + metrics: + - type: map + value: 49.19552407604654 + - type: mrr + value: 49.95269130379425 + - task: + type: Summarization + dataset: + type: mteb/summeval + name: MTEB SummEval + config: default + split: test + revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c + metrics: + - type: cos_sim_pearson + value: 29.345293033095427 + - type: cos_sim_spearman + value: 29.976931423258403 + - type: dot_pearson + value: 27.047078008958408 + - type: dot_spearman + value: 27.75894368380218 + - task: + type: Retrieval + dataset: + type: trec-covid + name: MTEB TRECCOVID + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 0.22 + - type: map_at_10 + value: 1.706 + - type: map_at_100 + value: 9.634 + - type: map_at_1000 + value: 23.665 + - type: map_at_3 + value: 0.5950000000000001 + - type: map_at_5 + value: 0.95 + - type: mrr_at_1 + value: 86.0 + - type: mrr_at_10 + value: 91.8 + - type: mrr_at_100 + value: 91.8 + - type: mrr_at_1000 + value: 91.8 + - type: mrr_at_3 + value: 91.0 + - type: mrr_at_5 + value: 91.8 + - type: ndcg_at_1 + value: 80.0 + - type: ndcg_at_10 + value: 72.573 + - type: ndcg_at_100 + value: 53.954 + - type: ndcg_at_1000 + value: 47.760999999999996 + - type: ndcg_at_3 + value: 76.173 + - type: ndcg_at_5 + value: 75.264 + - type: precision_at_1 + value: 86.0 + - type: precision_at_10 + value: 76.4 + - type: precision_at_100 + value: 55.50000000000001 + - type: precision_at_1000 + value: 21.802 + - type: precision_at_3 + value: 81.333 + - type: precision_at_5 + value: 80.4 + - type: recall_at_1 + value: 0.22 + - type: recall_at_10 + value: 1.925 + - type: recall_at_100 + value: 12.762 + - type: recall_at_1000 + value: 44.946000000000005 + - type: recall_at_3 + value: 0.634 + - type: recall_at_5 + value: 1.051 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (sqi-eng) + config: sqi-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 91.0 + - type: f1 + value: 88.55666666666666 + - type: precision + value: 87.46166666666667 + - type: recall + value: 91.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fry-eng) + config: fry-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 57.22543352601156 + - type: f1 + value: 51.03220478943021 + - type: precision + value: 48.8150289017341 + - type: recall + value: 57.22543352601156 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kur-eng) + config: kur-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 46.58536585365854 + - type: f1 + value: 39.66870798578116 + - type: precision + value: 37.416085946573745 + - type: recall + value: 46.58536585365854 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tur-eng) + config: tur-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 89.7 + - type: f1 + value: 86.77999999999999 + - type: precision + value: 85.45333333333332 + - type: recall + value: 89.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (deu-eng) + config: deu-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 97.39999999999999 + - type: f1 + value: 96.58333333333331 + - type: precision + value: 96.2 + - type: recall + value: 97.39999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nld-eng) + config: nld-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 92.4 + - type: f1 + value: 90.3 + - type: precision + value: 89.31666666666668 + - type: recall + value: 92.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ron-eng) + config: ron-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 86.9 + - type: f1 + value: 83.67190476190476 + - type: precision + value: 82.23333333333332 + - type: recall + value: 86.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ang-eng) + config: ang-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 50.0 + - type: f1 + value: 42.23229092632078 + - type: precision + value: 39.851634683724235 + - type: recall + value: 50.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ido-eng) + config: ido-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 76.3 + - type: f1 + value: 70.86190476190477 + - type: precision + value: 68.68777777777777 + - type: recall + value: 76.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (jav-eng) + config: jav-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 57.073170731707314 + - type: f1 + value: 50.658958927251604 + - type: precision + value: 48.26480836236933 + - type: recall + value: 57.073170731707314 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (isl-eng) + config: isl-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 68.2 + - type: f1 + value: 62.156507936507936 + - type: precision + value: 59.84964285714286 + - type: recall + value: 68.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (slv-eng) + config: slv-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 77.52126366950182 + - type: f1 + value: 72.8496210148701 + - type: precision + value: 70.92171498003819 + - type: recall + value: 77.52126366950182 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cym-eng) + config: cym-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 70.78260869565217 + - type: f1 + value: 65.32422360248447 + - type: precision + value: 63.063067367415194 + - type: recall + value: 70.78260869565217 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kaz-eng) + config: kaz-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 78.43478260869566 + - type: f1 + value: 73.02608695652172 + - type: precision + value: 70.63768115942028 + - type: recall + value: 78.43478260869566 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (est-eng) + config: est-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 60.9 + - type: f1 + value: 55.309753694581275 + - type: precision + value: 53.130476190476195 + - type: recall + value: 60.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (heb-eng) + config: heb-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 72.89999999999999 + - type: f1 + value: 67.92023809523809 + - type: precision + value: 65.82595238095237 + - type: recall + value: 72.89999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (gla-eng) + config: gla-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 46.80337756332931 + - type: f1 + value: 39.42174900558496 + - type: precision + value: 36.97101116280851 + - type: recall + value: 46.80337756332931 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mar-eng) + config: mar-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 89.8 + - type: f1 + value: 86.79 + - type: precision + value: 85.375 + - type: recall + value: 89.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lat-eng) + config: lat-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 47.199999999999996 + - type: f1 + value: 39.95484348984349 + - type: precision + value: 37.561071428571424 + - type: recall + value: 47.199999999999996 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bel-eng) + config: bel-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 87.8 + - type: f1 + value: 84.68190476190475 + - type: precision + value: 83.275 + - type: recall + value: 87.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pms-eng) + config: pms-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 48.76190476190476 + - type: f1 + value: 42.14965986394558 + - type: precision + value: 39.96743626743626 + - type: recall + value: 48.76190476190476 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (gle-eng) + config: gle-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 66.10000000000001 + - type: f1 + value: 59.58580086580086 + - type: precision + value: 57.150238095238095 + - type: recall + value: 66.10000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pes-eng) + config: pes-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 87.3 + - type: f1 + value: 84.0 + - type: precision + value: 82.48666666666666 + - type: recall + value: 87.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nob-eng) + config: nob-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 90.4 + - type: f1 + value: 87.79523809523809 + - type: precision + value: 86.6 + - type: recall + value: 90.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bul-eng) + config: bul-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 87.0 + - type: f1 + value: 83.81 + - type: precision + value: 82.36666666666666 + - type: recall + value: 87.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cbk-eng) + config: cbk-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 63.9 + - type: f1 + value: 57.76533189033189 + - type: precision + value: 55.50595238095239 + - type: recall + value: 63.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hun-eng) + config: hun-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 76.1 + - type: f1 + value: 71.83690476190478 + - type: precision + value: 70.04928571428573 + - type: recall + value: 76.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (uig-eng) + config: uig-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 66.3 + - type: f1 + value: 59.32626984126984 + - type: precision + value: 56.62535714285713 + - type: recall + value: 66.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (rus-eng) + config: rus-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 90.60000000000001 + - type: f1 + value: 87.96333333333334 + - type: precision + value: 86.73333333333333 + - type: recall + value: 90.60000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (spa-eng) + config: spa-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 93.10000000000001 + - type: f1 + value: 91.10000000000001 + - type: precision + value: 90.16666666666666 + - type: recall + value: 93.10000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hye-eng) + config: hye-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 85.71428571428571 + - type: f1 + value: 82.29142600436403 + - type: precision + value: 80.8076626877166 + - type: recall + value: 85.71428571428571 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tel-eng) + config: tel-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 88.88888888888889 + - type: f1 + value: 85.7834757834758 + - type: precision + value: 84.43732193732193 + - type: recall + value: 88.88888888888889 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (afr-eng) + config: afr-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 88.5 + - type: f1 + value: 85.67190476190476 + - type: precision + value: 84.43333333333332 + - type: recall + value: 88.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mon-eng) + config: mon-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 82.72727272727273 + - type: f1 + value: 78.21969696969695 + - type: precision + value: 76.18181818181819 + - type: recall + value: 82.72727272727273 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (arz-eng) + config: arz-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 61.0062893081761 + - type: f1 + value: 55.13976240391334 + - type: precision + value: 52.92112499659669 + - type: recall + value: 61.0062893081761 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hrv-eng) + config: hrv-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 89.5 + - type: f1 + value: 86.86666666666666 + - type: precision + value: 85.69166666666668 + - type: recall + value: 89.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nov-eng) + config: nov-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 73.54085603112841 + - type: f1 + value: 68.56031128404669 + - type: precision + value: 66.53047989623866 + - type: recall + value: 73.54085603112841 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (gsw-eng) + config: gsw-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 43.58974358974359 + - type: f1 + value: 36.45299145299145 + - type: precision + value: 33.81155881155882 + - type: recall + value: 43.58974358974359 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nds-eng) + config: nds-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 59.599999999999994 + - type: f1 + value: 53.264689754689755 + - type: precision + value: 50.869166666666665 + - type: recall + value: 59.599999999999994 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ukr-eng) + config: ukr-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 85.2 + - type: f1 + value: 81.61666666666665 + - type: precision + value: 80.02833333333335 + - type: recall + value: 85.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (uzb-eng) + config: uzb-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 63.78504672897196 + - type: f1 + value: 58.00029669188548 + - type: precision + value: 55.815809968847354 + - type: recall + value: 63.78504672897196 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lit-eng) + config: lit-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 66.5 + - type: f1 + value: 61.518333333333345 + - type: precision + value: 59.622363699102834 + - type: recall + value: 66.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ina-eng) + config: ina-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 88.6 + - type: f1 + value: 85.60222222222221 + - type: precision + value: 84.27916666666665 + - type: recall + value: 88.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lfn-eng) + config: lfn-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 58.699999999999996 + - type: f1 + value: 52.732375957375965 + - type: precision + value: 50.63214035964035 + - type: recall + value: 58.699999999999996 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (zsm-eng) + config: zsm-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 92.10000000000001 + - type: f1 + value: 89.99666666666667 + - type: precision + value: 89.03333333333333 + - type: recall + value: 92.10000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ita-eng) + config: ita-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 90.10000000000001 + - type: f1 + value: 87.55666666666667 + - type: precision + value: 86.36166666666668 + - type: recall + value: 90.10000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cmn-eng) + config: cmn-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 91.4 + - type: f1 + value: 88.89000000000001 + - type: precision + value: 87.71166666666666 + - type: recall + value: 91.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (lvs-eng) + config: lvs-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 65.7 + - type: f1 + value: 60.67427750410509 + - type: precision + value: 58.71785714285714 + - type: recall + value: 65.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (glg-eng) + config: glg-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 85.39999999999999 + - type: f1 + value: 81.93190476190475 + - type: precision + value: 80.37833333333333 + - type: recall + value: 85.39999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ceb-eng) + config: ceb-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 47.833333333333336 + - type: f1 + value: 42.006625781625786 + - type: precision + value: 40.077380952380956 + - type: recall + value: 47.833333333333336 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bre-eng) + config: bre-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 10.4 + - type: f1 + value: 8.24465007215007 + - type: precision + value: 7.664597069597071 + - type: recall + value: 10.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ben-eng) + config: ben-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 82.6 + - type: f1 + value: 77.76333333333334 + - type: precision + value: 75.57833333333332 + - type: recall + value: 82.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (swg-eng) + config: swg-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 52.67857142857143 + - type: f1 + value: 44.302721088435376 + - type: precision + value: 41.49801587301587 + - type: recall + value: 52.67857142857143 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (arq-eng) + config: arq-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 28.3205268935236 + - type: f1 + value: 22.426666605171157 + - type: precision + value: 20.685900116470915 + - type: recall + value: 28.3205268935236 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kab-eng) + config: kab-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 22.7 + - type: f1 + value: 17.833970473970474 + - type: precision + value: 16.407335164835164 + - type: recall + value: 22.7 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fra-eng) + config: fra-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 92.2 + - type: f1 + value: 89.92999999999999 + - type: precision + value: 88.87 + - type: recall + value: 92.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (por-eng) + config: por-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 91.4 + - type: f1 + value: 89.25 + - type: precision + value: 88.21666666666667 + - type: recall + value: 91.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tat-eng) + config: tat-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 69.19999999999999 + - type: f1 + value: 63.38269841269841 + - type: precision + value: 61.14773809523809 + - type: recall + value: 69.19999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (oci-eng) + config: oci-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 48.8 + - type: f1 + value: 42.839915639915645 + - type: precision + value: 40.770287114845935 + - type: recall + value: 48.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pol-eng) + config: pol-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 88.8 + - type: f1 + value: 85.90666666666668 + - type: precision + value: 84.54166666666666 + - type: recall + value: 88.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (war-eng) + config: war-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 46.6 + - type: f1 + value: 40.85892920804686 + - type: precision + value: 38.838223114604695 + - type: recall + value: 46.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (aze-eng) + config: aze-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 84.0 + - type: f1 + value: 80.14190476190475 + - type: precision + value: 78.45333333333333 + - type: recall + value: 84.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (vie-eng) + config: vie-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 90.5 + - type: f1 + value: 87.78333333333333 + - type: precision + value: 86.5 + - type: recall + value: 90.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (nno-eng) + config: nno-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 74.5 + - type: f1 + value: 69.48397546897547 + - type: precision + value: 67.51869047619049 + - type: recall + value: 74.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cha-eng) + config: cha-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 32.846715328467155 + - type: f1 + value: 27.828177499710343 + - type: precision + value: 26.63451511991658 + - type: recall + value: 32.846715328467155 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mhr-eng) + config: mhr-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 8.0 + - type: f1 + value: 6.07664116764988 + - type: precision + value: 5.544177607179943 + - type: recall + value: 8.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (dan-eng) + config: dan-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 87.6 + - type: f1 + value: 84.38555555555554 + - type: precision + value: 82.91583333333334 + - type: recall + value: 87.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ell-eng) + config: ell-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 87.5 + - type: f1 + value: 84.08333333333331 + - type: precision + value: 82.47333333333333 + - type: recall + value: 87.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (amh-eng) + config: amh-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 80.95238095238095 + - type: f1 + value: 76.13095238095238 + - type: precision + value: 74.05753968253967 + - type: recall + value: 80.95238095238095 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (pam-eng) + config: pam-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 8.799999999999999 + - type: f1 + value: 6.971422975172975 + - type: precision + value: 6.557814916172301 + - type: recall + value: 8.799999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hsb-eng) + config: hsb-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 44.099378881987576 + - type: f1 + value: 37.01649742022413 + - type: precision + value: 34.69420618488942 + - type: recall + value: 44.099378881987576 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (srp-eng) + config: srp-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 84.3 + - type: f1 + value: 80.32666666666667 + - type: precision + value: 78.60666666666665 + - type: recall + value: 84.3 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (epo-eng) + config: epo-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 92.5 + - type: f1 + value: 90.49666666666666 + - type: precision + value: 89.56666666666668 + - type: recall + value: 92.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kzj-eng) + config: kzj-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 10.0 + - type: f1 + value: 8.268423529875141 + - type: precision + value: 7.878118605532398 + - type: recall + value: 10.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (awa-eng) + config: awa-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 79.22077922077922 + - type: f1 + value: 74.27128427128426 + - type: precision + value: 72.28715728715729 + - type: recall + value: 79.22077922077922 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fao-eng) + config: fao-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 65.64885496183206 + - type: f1 + value: 58.87495456197747 + - type: precision + value: 55.992366412213734 + - type: recall + value: 65.64885496183206 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mal-eng) + config: mal-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 96.06986899563319 + - type: f1 + value: 94.78408539543909 + - type: precision + value: 94.15332362930616 + - type: recall + value: 96.06986899563319 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ile-eng) + config: ile-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 77.2 + - type: f1 + value: 71.72571428571428 + - type: precision + value: 69.41000000000001 + - type: recall + value: 77.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (bos-eng) + config: bos-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 86.4406779661017 + - type: f1 + value: 83.2391713747646 + - type: precision + value: 81.74199623352166 + - type: recall + value: 86.4406779661017 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cor-eng) + config: cor-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 8.4 + - type: f1 + value: 6.017828743398003 + - type: precision + value: 5.4829865484756795 + - type: recall + value: 8.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (cat-eng) + config: cat-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 83.5 + - type: f1 + value: 79.74833333333333 + - type: precision + value: 78.04837662337664 + - type: recall + value: 83.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (eus-eng) + config: eus-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 60.4 + - type: f1 + value: 54.467301587301584 + - type: precision + value: 52.23242424242424 + - type: recall + value: 60.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (yue-eng) + config: yue-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 74.9 + - type: f1 + value: 69.68699134199134 + - type: precision + value: 67.59873015873016 + - type: recall + value: 74.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (swe-eng) + config: swe-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 88.0 + - type: f1 + value: 84.9652380952381 + - type: precision + value: 83.66166666666666 + - type: recall + value: 88.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (dtp-eng) + config: dtp-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 9.1 + - type: f1 + value: 7.681244588744588 + - type: precision + value: 7.370043290043291 + - type: recall + value: 9.1 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kat-eng) + config: kat-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 80.9651474530831 + - type: f1 + value: 76.84220605132133 + - type: precision + value: 75.19606398962966 + - type: recall + value: 80.9651474530831 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (jpn-eng) + config: jpn-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 86.9 + - type: f1 + value: 83.705 + - type: precision + value: 82.3120634920635 + - type: recall + value: 86.9 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (csb-eng) + config: csb-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 29.64426877470356 + - type: f1 + value: 23.98763072676116 + - type: precision + value: 22.506399397703746 + - type: recall + value: 29.64426877470356 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (xho-eng) + config: xho-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 70.4225352112676 + - type: f1 + value: 62.84037558685445 + - type: precision + value: 59.56572769953053 + - type: recall + value: 70.4225352112676 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (orv-eng) + config: orv-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 19.64071856287425 + - type: f1 + value: 15.125271011207756 + - type: precision + value: 13.865019261197494 + - type: recall + value: 19.64071856287425 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ind-eng) + config: ind-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 90.2 + - type: f1 + value: 87.80666666666666 + - type: precision + value: 86.70833333333331 + - type: recall + value: 90.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tuk-eng) + config: tuk-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 23.15270935960591 + - type: f1 + value: 18.407224958949097 + - type: precision + value: 16.982385430661292 + - type: recall + value: 23.15270935960591 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (max-eng) + config: max-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 55.98591549295775 + - type: f1 + value: 49.94718309859154 + - type: precision + value: 47.77864154624717 + - type: recall + value: 55.98591549295775 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (swh-eng) + config: swh-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 73.07692307692307 + - type: f1 + value: 66.74358974358974 + - type: precision + value: 64.06837606837607 + - type: recall + value: 73.07692307692307 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (hin-eng) + config: hin-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 94.89999999999999 + - type: f1 + value: 93.25 + - type: precision + value: 92.43333333333332 + - type: recall + value: 94.89999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (dsb-eng) + config: dsb-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 37.78705636743215 + - type: f1 + value: 31.63899658680452 + - type: precision + value: 29.72264397629742 + - type: recall + value: 37.78705636743215 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ber-eng) + config: ber-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 21.6 + - type: f1 + value: 16.91697302697303 + - type: precision + value: 15.71225147075147 + - type: recall + value: 21.6 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tam-eng) + config: tam-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 85.01628664495115 + - type: f1 + value: 81.38514037536838 + - type: precision + value: 79.83170466883823 + - type: recall + value: 85.01628664495115 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (slk-eng) + config: slk-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 83.39999999999999 + - type: f1 + value: 79.96380952380952 + - type: precision + value: 78.48333333333333 + - type: recall + value: 83.39999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tgl-eng) + config: tgl-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 83.2 + - type: f1 + value: 79.26190476190476 + - type: precision + value: 77.58833333333334 + - type: recall + value: 83.2 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ast-eng) + config: ast-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 75.59055118110236 + - type: f1 + value: 71.66854143232096 + - type: precision + value: 70.30183727034121 + - type: recall + value: 75.59055118110236 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (mkd-eng) + config: mkd-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 65.5 + - type: f1 + value: 59.26095238095238 + - type: precision + value: 56.81909090909092 + - type: recall + value: 65.5 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (khm-eng) + config: khm-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 55.26315789473685 + - type: f1 + value: 47.986523325858506 + - type: precision + value: 45.33950006595436 + - type: recall + value: 55.26315789473685 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ces-eng) + config: ces-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 82.89999999999999 + - type: f1 + value: 78.835 + - type: precision + value: 77.04761904761905 + - type: recall + value: 82.89999999999999 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tzl-eng) + config: tzl-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 43.269230769230774 + - type: f1 + value: 36.20421245421245 + - type: precision + value: 33.57371794871795 + - type: recall + value: 43.269230769230774 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (urd-eng) + config: urd-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 88.0 + - type: f1 + value: 84.70666666666666 + - type: precision + value: 83.23166666666665 + - type: recall + value: 88.0 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (ara-eng) + config: ara-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 77.4 + - type: f1 + value: 72.54666666666667 + - type: precision + value: 70.54318181818181 + - type: recall + value: 77.4 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (kor-eng) + config: kor-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 78.60000000000001 + - type: f1 + value: 74.1588888888889 + - type: precision + value: 72.30250000000001 + - type: recall + value: 78.60000000000001 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (yid-eng) + config: yid-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 72.40566037735849 + - type: f1 + value: 66.82587328813744 + - type: precision + value: 64.75039308176099 + - type: recall + value: 72.40566037735849 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (fin-eng) + config: fin-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 73.8 + - type: f1 + value: 68.56357142857144 + - type: precision + value: 66.3178822055138 + - type: recall + value: 73.8 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (tha-eng) + config: tha-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 91.78832116788321 + - type: f1 + value: 89.3552311435523 + - type: precision + value: 88.20559610705597 + - type: recall + value: 91.78832116788321 + - task: + type: BitextMining + dataset: + type: mteb/tatoeba-bitext-mining + name: MTEB Tatoeba (wuu-eng) + config: wuu-eng + split: test + revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 + metrics: + - type: accuracy + value: 74.3 + - type: f1 + value: 69.05085581085581 + - type: precision + value: 66.955 + - type: recall + value: 74.3 + - task: + type: Retrieval + dataset: + type: webis-touche2020 + name: MTEB Touche2020 + config: default + split: test + revision: None + metrics: + - type: map_at_1 + value: 2.896 + - type: map_at_10 + value: 8.993 + - type: map_at_100 + value: 14.133999999999999 + - type: map_at_1000 + value: 15.668000000000001 + - type: map_at_3 + value: 5.862 + - type: map_at_5 + value: 7.17 + - type: mrr_at_1 + value: 34.694 + - type: mrr_at_10 + value: 42.931000000000004 + - type: mrr_at_100 + value: 44.81 + - type: mrr_at_1000 + value: 44.81 + - type: mrr_at_3 + value: 38.435 + - type: mrr_at_5 + value: 41.701 + - type: ndcg_at_1 + value: 31.633 + - type: ndcg_at_10 + value: 21.163 + - type: ndcg_at_100 + value: 33.306000000000004 + - type: ndcg_at_1000 + value: 45.275999999999996 + - type: ndcg_at_3 + value: 25.685999999999996 + - type: ndcg_at_5 + value: 23.732 + - type: precision_at_1 + value: 34.694 + - type: precision_at_10 + value: 17.755000000000003 + - type: precision_at_100 + value: 6.938999999999999 + - type: precision_at_1000 + value: 1.48 + - type: precision_at_3 + value: 25.85 + - type: precision_at_5 + value: 23.265 + - type: recall_at_1 + value: 2.896 + - type: recall_at_10 + value: 13.333999999999998 + - type: recall_at_100 + value: 43.517 + - type: recall_at_1000 + value: 79.836 + - type: recall_at_3 + value: 6.306000000000001 + - type: recall_at_5 + value: 8.825 + - task: + type: Classification + dataset: + type: mteb/toxic_conversations_50k + name: MTEB ToxicConversationsClassification + config: default + split: test + revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c + metrics: + - type: accuracy + value: 69.3874 + - type: ap + value: 13.829909072469423 + - type: f1 + value: 53.54534203543492 + - task: + type: Classification + dataset: + type: mteb/tweet_sentiment_extraction + name: MTEB TweetSentimentExtractionClassification + config: default + split: test + revision: d604517c81ca91fe16a244d1248fc021f9ecee7a + metrics: + - type: accuracy + value: 62.62026032823995 + - type: f1 + value: 62.85251350485221 + - task: + type: Clustering + dataset: + type: mteb/twentynewsgroups-clustering + name: MTEB TwentyNewsgroupsClustering + config: default + split: test + revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 + metrics: + - type: v_measure + value: 33.21527881409797 + - task: + type: PairClassification + dataset: + type: mteb/twittersemeval2015-pairclassification + name: MTEB TwitterSemEval2015 + config: default + split: test + revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 + metrics: + - type: cos_sim_accuracy + value: 84.97943613280086 + - type: cos_sim_ap + value: 70.75454316885921 + - type: cos_sim_f1 + value: 65.38274012676743 + - type: cos_sim_precision + value: 60.761214318078835 + - type: cos_sim_recall + value: 70.76517150395777 + - type: dot_accuracy + value: 79.0546581629612 + - type: dot_ap + value: 47.3197121792147 + - type: dot_f1 + value: 49.20106524633821 + - type: dot_precision + value: 42.45499808502489 + - type: dot_recall + value: 58.49604221635884 + - type: euclidean_accuracy + value: 85.08076533349228 + - type: euclidean_ap + value: 70.95016106374474 + - type: euclidean_f1 + value: 65.43987900176455 + - type: euclidean_precision + value: 62.64478764478765 + - type: euclidean_recall + value: 68.49604221635884 + - type: manhattan_accuracy + value: 84.93771234428085 + - type: manhattan_ap + value: 70.63668388755362 + - type: manhattan_f1 + value: 65.23895401262398 + - type: manhattan_precision + value: 56.946084218811485 + - type: manhattan_recall + value: 76.35883905013192 + - type: max_accuracy + value: 85.08076533349228 + - type: max_ap + value: 70.95016106374474 + - type: max_f1 + value: 65.43987900176455 + - task: + type: PairClassification + dataset: + type: mteb/twitterurlcorpus-pairclassification + name: MTEB TwitterURLCorpus + config: default + split: test + revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf + metrics: + - type: cos_sim_accuracy + value: 88.69096130709822 + - type: cos_sim_ap + value: 84.82526278228542 + - type: cos_sim_f1 + value: 77.65485060585536 + - type: cos_sim_precision + value: 75.94582658619167 + - type: cos_sim_recall + value: 79.44256236526024 + - type: dot_accuracy + value: 80.97954748321496 + - type: dot_ap + value: 64.81642914145866 + - type: dot_f1 + value: 60.631996987229975 + - type: dot_precision + value: 54.5897293631712 + - type: dot_recall + value: 68.17831844779796 + - type: euclidean_accuracy + value: 88.6987231730508 + - type: euclidean_ap + value: 84.80003825477253 + - type: euclidean_f1 + value: 77.67194179854496 + - type: euclidean_precision + value: 75.7128235122094 + - type: euclidean_recall + value: 79.73514012935017 + - type: manhattan_accuracy + value: 88.62692591298949 + - type: manhattan_ap + value: 84.80451408255276 + - type: manhattan_f1 + value: 77.69888949572183 + - type: manhattan_precision + value: 73.70311528631622 + - type: manhattan_recall + value: 82.15275639051433 + - type: max_accuracy + value: 88.6987231730508 + - type: max_ap + value: 84.82526278228542 + - type: max_f1 + value: 77.69888949572183 +language: +- multilingual +- af +- am +- ar +- as +- az +- be +- bg +- bn +- br +- bs +- ca +- cs +- cy +- da +- de +- el +- en +- eo +- es +- et +- eu +- fa +- fi +- fr +- fy +- ga +- gd +- gl +- gu +- ha +- he +- hi +- hr +- hu +- hy +- id +- is +- it +- ja +- jv +- ka +- kk +- km +- kn +- ko +- ku +- ky +- la +- lo +- lt +- lv +- mg +- mk +- ml +- mn +- mr +- ms +- my +- ne +- nl +- 'no' +- om +- or +- pa +- pl +- ps +- pt +- ro +- ru +- sa +- sd +- si +- sk +- sl +- so +- sq +- sr +- su +- sv +- sw +- ta +- te +- th +- tl +- tr +- ug +- uk +- ur +- uz +- vi +- xh +- yi +- zh +license: mit +--- + +## Multilingual-E5-small + +[Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://arxiv.org/pdf/2212.03533.pdf). +Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022 + +This model has 12 layers and the embedding size is 384. + +## Usage + +Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset. + +```python +import torch.nn.functional as F + +from torch import Tensor +from transformers import AutoTokenizer, AutoModel + + +def average_pool(last_hidden_states: Tensor, + attention_mask: Tensor) -> Tensor: + last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0) + return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None] + + +# Each input text should start with "query: " or "passage: ", even for non-English texts. +# For tasks other than retrieval, you can simply use the "query: " prefix. +input_texts = ['query: how much protein should a female eat', + 'query: 南瓜的家常做法', + "passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.", + "passage: 1.清炒南瓜丝 原料:嫩南瓜半个 调料:葱、盐、白糖、鸡精 做法: 1、南瓜用刀薄薄的削去表面一层皮,用勺子刮去瓤 2、擦成细丝(没有擦菜板就用刀慢慢切成细丝) 3、锅烧热放油,入葱花煸出香味 4、入南瓜丝快速翻炒一分钟左右,放盐、一点白糖和鸡精调味出锅 2.香葱炒南瓜 原料:南瓜1只 调料:香葱、蒜末、橄榄油、盐 做法: 1、将南瓜去皮,切成片 2、油锅8成热后,将蒜末放入爆香 3、爆香后,将南瓜片放入,翻炒 4、在翻炒的同时,可以不时地往锅里加水,但不要太多 5、放入盐,炒匀 6、南瓜差不多软和绵了之后,就可以关火 7、撒入香葱,即可出锅"] + +tokenizer = AutoTokenizer.from_pretrained('intfloat/multilingual-e5-small') +model = AutoModel.from_pretrained('intfloat/multilingual-e5-small') + +# Tokenize the input texts +batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt') + +outputs = model(**batch_dict) +embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask']) + +# (Optionally) normalize embeddings +embeddings = F.normalize(embeddings, p=2, dim=1) +scores = (embeddings[:2] @ embeddings[2:].T) * 100 +print(scores.tolist()) +``` + +## Supported Languages + +This model is initialized from [microsoft/Multilingual-MiniLM-L12-H384](https://huggingface.co/microsoft/Multilingual-MiniLM-L12-H384) +and continually trained on a mixture of multilingual datasets. +It supports 100 languages from xlm-roberta, +but low-resource languages may see performance degradation. + +## Training Details + +**Initialization**: [microsoft/Multilingual-MiniLM-L12-H384](https://huggingface.co/microsoft/Multilingual-MiniLM-L12-H384) + +**First stage**: contrastive pre-training with weak supervision + +| Dataset | Weak supervision | # of text pairs | +|--------------------------------------------------------------------------------------------------------|---------------------------------------|-----------------| +| Filtered [mC4](https://huggingface.co/datasets/mc4) | (title, page content) | 1B | +| [CC News](https://huggingface.co/datasets/intfloat/multilingual_cc_news) | (title, news content) | 400M | +| [NLLB](https://huggingface.co/datasets/allenai/nllb) | translation pairs | 2.4B | +| [Wikipedia](https://huggingface.co/datasets/intfloat/wikipedia) | (hierarchical section title, passage) | 150M | +| Filtered [Reddit](https://www.reddit.com/) | (comment, response) | 800M | +| [S2ORC](https://github.com/allenai/s2orc) | (title, abstract) and citation pairs | 100M | +| [Stackexchange](https://stackexchange.com/) | (question, answer) | 50M | +| [xP3](https://huggingface.co/datasets/bigscience/xP3) | (input prompt, response) | 80M | +| [Miscellaneous unsupervised SBERT data](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | - | 10M | + +**Second stage**: supervised fine-tuning + +| Dataset | Language | # of text pairs | +|----------------------------------------------------------------------------------------|--------------|-----------------| +| [MS MARCO](https://microsoft.github.io/msmarco/) | English | 500k | +| [NQ](https://github.com/facebookresearch/DPR) | English | 70k | +| [Trivia QA](https://github.com/facebookresearch/DPR) | English | 60k | +| [NLI from SimCSE](https://github.com/princeton-nlp/SimCSE) | English | <300k | +| [ELI5](https://huggingface.co/datasets/eli5) | English | 500k | +| [DuReader Retrieval](https://github.com/baidu/DuReader/tree/master/DuReader-Retrieval) | Chinese | 86k | +| [KILT Fever](https://huggingface.co/datasets/kilt_tasks) | English | 70k | +| [KILT HotpotQA](https://huggingface.co/datasets/kilt_tasks) | English | 70k | +| [SQuAD](https://huggingface.co/datasets/squad) | English | 87k | +| [Quora](https://huggingface.co/datasets/quora) | English | 150k | +| [Mr. TyDi](https://huggingface.co/datasets/castorini/mr-tydi) | 11 languages | 50k | +| [MIRACL](https://huggingface.co/datasets/miracl/miracl) | 16 languages | 40k | + +For all labeled datasets, we only use its training set for fine-tuning. + +For other training details, please refer to our paper at [https://arxiv.org/pdf/2212.03533.pdf](https://arxiv.org/pdf/2212.03533.pdf). + +## Benchmark Evaluation + +Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results +on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316). + +## Citation + +If you find our paper or models helpful, please consider cite as follows: + +``` +@article{wang2022text, + title={Text Embeddings by Weakly-Supervised Contrastive Pre-training}, + author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu}, + journal={arXiv preprint arXiv:2212.03533}, + year={2022} +} +``` + +## Limitations + +Long texts will be truncated to at most 512 tokens. +