--- license: cc-by-nc-4.0 tags: - mteb model-index: - name: text_sonar_basic_encoder results: - task: type: Clustering dataset: type: PL-MTEB/8tags-clustering name: MTEB 8TagsClustering config: default split: test revision: None metrics: - type: v_measure value: 14.51482032569021 - task: type: STS dataset: type: C-MTEB/AFQMC name: MTEB AFQMC config: default split: validation revision: b44c3b011063adb25877c13823db83bb193913c4 metrics: - type: cos_sim_pearson value: 17.970264353352682 - type: cos_sim_spearman value: 17.633997882973155 - type: euclidean_pearson value: 14.014776236053123 - type: euclidean_spearman value: 15.28941515698961 - type: manhattan_pearson value: 13.891563198299256 - type: manhattan_spearman value: 15.158415586569143 - task: type: STS dataset: type: C-MTEB/ATEC name: MTEB ATEC config: default split: test revision: 0f319b1142f28d00e055a6770f3f726ae9b7d865 metrics: - type: cos_sim_pearson value: 27.670887710562912 - type: cos_sim_spearman value: 26.176635036642804 - type: euclidean_pearson value: 24.115430353423346 - type: euclidean_spearman value: 24.31920807107195 - type: manhattan_pearson value: 23.998151286247396 - type: manhattan_spearman value: 24.167187716649224 - task: type: Classification dataset: type: PL-MTEB/allegro-reviews name: MTEB AllegroReviews config: default split: test revision: None metrics: - type: accuracy value: 29.85089463220676 - type: f1 value: 27.632986641843516 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en) config: en split: test revision: e8379541af4e31359cca9fbcf4b00f2671dba205 metrics: - type: accuracy value: 78.98507462686568 - type: ap value: 44.08138737273602 - type: f1 value: 73.46285533200773 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (de) config: de split: test revision: e8379541af4e31359cca9fbcf4b00f2671dba205 metrics: - type: accuracy value: 72.00214132762312 - type: ap value: 82.19993680100252 - type: f1 value: 70.11447824125146 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (en-ext) config: en-ext split: test revision: e8379541af4e31359cca9fbcf4b00f2671dba205 metrics: - type: accuracy value: 81.28935532233884 - type: ap value: 30.144590364670528 - type: f1 value: 68.42566054607258 - task: type: Classification dataset: type: mteb/amazon_counterfactual name: MTEB AmazonCounterfactualClassification (ja) config: ja split: test revision: e8379541af4e31359cca9fbcf4b00f2671dba205 metrics: - type: accuracy value: 67.7730192719486 - type: ap value: 18.42471917082788 - type: f1 value: 55.72761001975526 - task: type: Classification dataset: type: mteb/amazon_polarity name: MTEB AmazonPolarityClassification config: default split: test revision: e2d317d38cd51312af73b3d32a06d1a08b442046 metrics: - type: accuracy value: 61.521125 - type: ap value: 57.23445163027374 - type: f1 value: 60.73008406893218 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (en) config: en split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 30.491999999999997 - type: f1 value: 29.557204623883994 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (de) config: de split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 33.386 - type: f1 value: 31.84048812033152 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (es) config: es split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 34.866 - type: f1 value: 32.99012859140412 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (fr) config: fr split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 34.062000000000005 - type: f1 value: 33.25848878845983 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (ja) config: ja split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 31.023999999999997 - type: f1 value: 30.458831589997278 - task: type: Classification dataset: type: mteb/amazon_reviews_multi name: MTEB AmazonReviewsClassification (zh) config: zh split: test revision: 1399c76144fd37290681b995c656ef9b2e06e26d metrics: - type: accuracy value: 30.994 - type: f1 value: 30.061575356482727 - task: type: Classification dataset: type: DDSC/angry-tweets name: MTEB AngryTweetsClassification config: default split: test revision: 20b0e6081892e78179356fada741b7afa381443d metrics: - type: accuracy value: 48.223495702005735 - type: f1 value: 47.162760921004896 - task: type: Retrieval dataset: type: arguana name: MTEB ArguAna config: default split: test revision: None metrics: - type: map_at_1 value: 8.677 - type: map_at_10 value: 14.732000000000001 - type: map_at_100 value: 15.501999999999999 - type: map_at_1000 value: 15.583 - type: map_at_3 value: 12.553 - type: map_at_5 value: 13.822999999999999 - type: mrr_at_1 value: 8.819 - type: mrr_at_10 value: 14.787 - type: mrr_at_100 value: 15.557000000000002 - type: mrr_at_1000 value: 15.638 - type: mrr_at_3 value: 12.648000000000001 - type: mrr_at_5 value: 13.879 - type: ndcg_at_1 value: 8.677 - type: ndcg_at_10 value: 18.295 - type: ndcg_at_100 value: 22.353 - type: ndcg_at_1000 value: 24.948999999999998 - type: ndcg_at_3 value: 13.789000000000001 - type: ndcg_at_5 value: 16.075 - type: precision_at_1 value: 8.677 - type: precision_at_10 value: 2.98 - type: precision_at_100 value: 0.49500000000000005 - type: precision_at_1000 value: 0.07100000000000001 - type: precision_at_3 value: 5.785 - type: precision_at_5 value: 4.58 - type: recall_at_1 value: 8.677 - type: recall_at_10 value: 29.801 - type: recall_at_100 value: 49.502 - type: recall_at_1000 value: 70.91 - type: recall_at_3 value: 17.354 - type: recall_at_5 value: 22.902 - task: type: Clustering dataset: type: mteb/arxiv-clustering-p2p name: MTEB ArxivClusteringP2P config: default split: test revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d metrics: - type: v_measure value: 25.78534535464707 - task: type: Clustering dataset: type: mteb/arxiv-clustering-s2s name: MTEB ArxivClusteringS2S config: default split: test revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 metrics: - type: v_measure value: 22.935310338963955 - task: type: Reranking dataset: type: mteb/askubuntudupquestions-reranking name: MTEB AskUbuntuDupQuestions config: default split: test revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 metrics: - type: map value: 50.88606427049027 - type: mrr value: 65.13004001231148 - task: type: STS dataset: type: mteb/biosses-sts name: MTEB BIOSSES config: default split: test revision: d3fb88f8f02e40887cd149695127462bbcf29b4a metrics: - type: cos_sim_pearson value: 77.15058570798328 - type: cos_sim_spearman value: 79.10541692841936 - type: euclidean_pearson value: 57.77256072255626 - type: euclidean_spearman value: 61.9206880133157 - type: manhattan_pearson value: 57.61971995547671 - type: manhattan_spearman value: 61.65983869619309 - task: type: STS dataset: type: C-MTEB/BQ name: MTEB BQ config: default split: test revision: e3dda5e115e487b39ec7e618c0c6a29137052a55 metrics: - type: cos_sim_pearson value: 37.84739182483413 - type: cos_sim_spearman value: 37.6627002178424 - type: euclidean_pearson value: 31.20207774134489 - type: euclidean_spearman value: 32.258185520047554 - type: manhattan_pearson value: 31.45043795745484 - type: manhattan_spearman value: 32.432706960003095 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (de-en) config: de-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 98.97703549060543 - type: f1 value: 98.82393876130828 - type: precision value: 98.74913013221992 - type: recall value: 98.97703549060543 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (fr-en) config: fr-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 98.34910851860005 - type: f1 value: 98.09487123046446 - type: precision value: 97.97032063981217 - type: recall value: 98.34910851860005 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (ru-en) config: ru-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 97.60304814686526 - type: f1 value: 97.36520032328832 - type: precision value: 97.24743101258517 - type: recall value: 97.60304814686526 - task: type: BitextMining dataset: type: mteb/bucc-bitext-mining name: MTEB BUCC (zh-en) config: zh-en split: test revision: d51519689f32196a32af33b075a01d0e7c51e252 metrics: - type: accuracy value: 98.78883622959452 - type: f1 value: 98.71862383710724 - type: precision value: 98.68351764086361 - type: recall value: 98.78883622959452 - task: type: Classification dataset: type: mteb/banking77 name: MTEB Banking77Classification config: default split: test revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 metrics: - type: accuracy value: 70.12012987012987 - type: f1 value: 69.2899512261214 - task: type: Clustering dataset: type: jinaai/big-patent-clustering name: MTEB BigPatentClustering config: default split: test revision: 62d5330920bca426ce9d3c76ea914f15fc83e891 metrics: - type: v_measure value: 7.691890146277049 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-p2p name: MTEB BiorxivClusteringP2P config: default split: test revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 metrics: - type: v_measure value: 18.50684170881271 - task: type: Clustering dataset: type: mteb/biorxiv-clustering-s2s name: MTEB BiorxivClusteringS2S config: default split: test revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 metrics: - type: v_measure value: 16.69474554114157 - task: type: Clustering dataset: type: slvnwhrl/blurbs-clustering-p2p name: MTEB BlurbsClusteringP2P config: default split: test revision: a2dd5b02a77de3466a3eaa98ae586b5610314496 metrics: - type: v_measure value: 25.00467105836009 - task: type: Clustering dataset: type: slvnwhrl/blurbs-clustering-s2s name: MTEB BlurbsClusteringS2S config: default split: test revision: 9bfff9a7f8f6dc6ffc9da71c48dd48b68696471d metrics: - type: v_measure value: 13.043482009500194 - task: type: BitextMining dataset: type: strombergnlp/bornholmsk_parallel name: MTEB BornholmBitextMining config: default split: test revision: 3bc5cfb4ec514264fe2db5615fac9016f7251552 metrics: - type: accuracy value: 54.0 - type: f1 value: 47.33693528693529 - type: precision value: 45.04007936507936 - type: recall value: 54.0 - task: type: Classification dataset: type: PL-MTEB/cbd name: MTEB CBD config: default split: test revision: None metrics: - type: accuracy value: 66.62 - type: ap value: 18.77536577595191 - type: f1 value: 54.33122331135114 - task: type: PairClassification dataset: type: PL-MTEB/cdsce-pairclassification name: MTEB CDSC-E config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 84.39999999999999 - type: cos_sim_ap value: 59.968589741258036 - type: cos_sim_f1 value: 54.90909090909091 - type: cos_sim_precision value: 41.94444444444444 - type: cos_sim_recall value: 79.47368421052632 - type: dot_accuracy value: 83.7 - type: dot_ap value: 51.707245136367455 - type: dot_f1 value: 52.03619909502262 - type: dot_precision value: 45.63492063492063 - type: dot_recall value: 60.526315789473685 - type: euclidean_accuracy value: 84.3 - type: euclidean_ap value: 60.640177471697974 - type: euclidean_f1 value: 56.55737704918033 - type: euclidean_precision value: 46.308724832214764 - type: euclidean_recall value: 72.63157894736842 - type: manhattan_accuracy value: 84.3 - type: manhattan_ap value: 60.642449283992626 - type: manhattan_f1 value: 56.82242990654205 - type: manhattan_precision value: 44.05797101449275 - type: manhattan_recall value: 80.0 - type: max_accuracy value: 84.39999999999999 - type: max_ap value: 60.642449283992626 - type: max_f1 value: 56.82242990654205 - task: type: STS dataset: type: PL-MTEB/cdscr-sts name: MTEB CDSC-R config: default split: test revision: None metrics: - type: cos_sim_pearson value: 83.84274209712814 - type: cos_sim_spearman value: 85.76929022706523 - type: euclidean_pearson value: 79.29678840568863 - type: euclidean_spearman value: 83.83915785622271 - type: manhattan_pearson value: 79.51377951133192 - type: manhattan_spearman value: 84.01330353535174 - task: type: Clustering dataset: type: C-MTEB/CLSClusteringP2P name: MTEB CLSClusteringP2P config: default split: test revision: 4b6227591c6c1a73bc76b1055f3b7f3588e72476 metrics: - type: v_measure value: 26.145917838254785 - task: type: Clustering dataset: type: C-MTEB/CLSClusteringS2S name: MTEB CLSClusteringS2S config: default split: test revision: e458b3f5414b62b7f9f83499ac1f5497ae2e869f metrics: - type: v_measure value: 28.0310744137802 - task: type: Reranking dataset: type: C-MTEB/CMedQAv1-reranking name: MTEB CMedQAv1 config: default split: test revision: 8d7f1e942507dac42dc58017c1a001c3717da7df metrics: - type: map value: 34.30811576227924 - type: mrr value: 40.25416666666666 - task: type: Reranking dataset: type: C-MTEB/CMedQAv2-reranking name: MTEB CMedQAv2 config: default split: test revision: 23d186750531a14a0357ca22cd92d712fd512ea0 metrics: - type: map value: 35.87700379259406 - type: mrr value: 40.80206349206349 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackAndroidRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 7.655000000000001 - type: map_at_10 value: 11.681999999999999 - type: map_at_100 value: 12.464 - type: map_at_1000 value: 12.603 - type: map_at_3 value: 10.514 - type: map_at_5 value: 11.083 - type: mrr_at_1 value: 10.157 - type: mrr_at_10 value: 14.773 - type: mrr_at_100 value: 15.581999999999999 - type: mrr_at_1000 value: 15.68 - type: mrr_at_3 value: 13.519 - type: mrr_at_5 value: 14.049 - type: ndcg_at_1 value: 10.157 - type: ndcg_at_10 value: 14.527999999999999 - type: ndcg_at_100 value: 18.695999999999998 - type: ndcg_at_1000 value: 22.709 - type: ndcg_at_3 value: 12.458 - type: ndcg_at_5 value: 13.152 - type: precision_at_1 value: 10.157 - type: precision_at_10 value: 2.976 - type: precision_at_100 value: 0.634 - type: precision_at_1000 value: 0.131 - type: precision_at_3 value: 6.152 - type: precision_at_5 value: 4.378 - type: recall_at_1 value: 7.655000000000001 - type: recall_at_10 value: 20.105 - type: recall_at_100 value: 39.181 - type: recall_at_1000 value: 68.06400000000001 - type: recall_at_3 value: 14.033000000000001 - type: recall_at_5 value: 16.209 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackEnglishRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 3.2329999999999997 - type: map_at_10 value: 5.378 - type: map_at_100 value: 5.774 - type: map_at_1000 value: 5.863 - type: map_at_3 value: 4.598 - type: map_at_5 value: 4.9750000000000005 - type: mrr_at_1 value: 4.076 - type: mrr_at_10 value: 6.679 - type: mrr_at_100 value: 7.151000000000001 - type: mrr_at_1000 value: 7.24 - type: mrr_at_3 value: 5.722 - type: mrr_at_5 value: 6.2059999999999995 - type: ndcg_at_1 value: 4.076 - type: ndcg_at_10 value: 6.994 - type: ndcg_at_100 value: 9.366 - type: ndcg_at_1000 value: 12.181000000000001 - type: ndcg_at_3 value: 5.356000000000001 - type: ndcg_at_5 value: 6.008 - type: precision_at_1 value: 4.076 - type: precision_at_10 value: 1.459 - type: precision_at_100 value: 0.334 - type: precision_at_1000 value: 0.075 - type: precision_at_3 value: 2.718 - type: precision_at_5 value: 2.089 - type: recall_at_1 value: 3.2329999999999997 - type: recall_at_10 value: 10.749 - type: recall_at_100 value: 21.776 - type: recall_at_1000 value: 42.278999999999996 - type: recall_at_3 value: 6.146999999999999 - type: recall_at_5 value: 7.779999999999999 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGamingRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 8.036 - type: map_at_10 value: 12.727 - type: map_at_100 value: 13.532 - type: map_at_1000 value: 13.653 - type: map_at_3 value: 11.15 - type: map_at_5 value: 11.965 - type: mrr_at_1 value: 9.404 - type: mrr_at_10 value: 14.493 - type: mrr_at_100 value: 15.274 - type: mrr_at_1000 value: 15.370000000000001 - type: mrr_at_3 value: 12.853 - type: mrr_at_5 value: 13.696 - type: ndcg_at_1 value: 9.404 - type: ndcg_at_10 value: 15.784 - type: ndcg_at_100 value: 20.104 - type: ndcg_at_1000 value: 23.357 - type: ndcg_at_3 value: 12.61 - type: ndcg_at_5 value: 13.988 - type: precision_at_1 value: 9.404 - type: precision_at_10 value: 2.947 - type: precision_at_100 value: 0.562 - type: precision_at_1000 value: 0.093 - type: precision_at_3 value: 6.04 - type: precision_at_5 value: 4.4639999999999995 - type: recall_at_1 value: 8.036 - type: recall_at_10 value: 23.429 - type: recall_at_100 value: 43.728 - type: recall_at_1000 value: 68.10000000000001 - type: recall_at_3 value: 14.99 - type: recall_at_5 value: 18.274 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackGisRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 3.653 - type: map_at_10 value: 5.941 - type: map_at_100 value: 6.512 - type: map_at_1000 value: 6.6129999999999995 - type: map_at_3 value: 5.2540000000000004 - type: map_at_5 value: 5.645 - type: mrr_at_1 value: 3.955 - type: mrr_at_10 value: 6.4079999999999995 - type: mrr_at_100 value: 7.005999999999999 - type: mrr_at_1000 value: 7.105 - type: mrr_at_3 value: 5.593 - type: mrr_at_5 value: 6.051 - type: ndcg_at_1 value: 3.955 - type: ndcg_at_10 value: 7.342 - type: ndcg_at_100 value: 10.543 - type: ndcg_at_1000 value: 14.011000000000001 - type: ndcg_at_3 value: 5.853 - type: ndcg_at_5 value: 6.586 - type: precision_at_1 value: 3.955 - type: precision_at_10 value: 1.266 - type: precision_at_100 value: 0.315 - type: precision_at_1000 value: 0.066 - type: precision_at_3 value: 2.5989999999999998 - type: precision_at_5 value: 1.966 - type: recall_at_1 value: 3.653 - type: recall_at_10 value: 11.232000000000001 - type: recall_at_100 value: 26.625 - type: recall_at_1000 value: 54.476 - type: recall_at_3 value: 7.269 - type: recall_at_5 value: 8.982999999999999 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackMathematicaRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 2.257 - type: map_at_10 value: 3.881 - type: map_at_100 value: 4.279 - type: map_at_1000 value: 4.417 - type: map_at_3 value: 3.4070000000000005 - type: map_at_5 value: 3.744 - type: mrr_at_1 value: 2.9850000000000003 - type: mrr_at_10 value: 4.756 - type: mrr_at_100 value: 5.228 - type: mrr_at_1000 value: 5.354 - type: mrr_at_3 value: 4.125 - type: mrr_at_5 value: 4.567 - type: ndcg_at_1 value: 2.9850000000000003 - type: ndcg_at_10 value: 4.936999999999999 - type: ndcg_at_100 value: 7.664 - type: ndcg_at_1000 value: 12.045 - type: ndcg_at_3 value: 3.956 - type: ndcg_at_5 value: 4.584 - type: precision_at_1 value: 2.9850000000000003 - type: precision_at_10 value: 0.9329999999999999 - type: precision_at_100 value: 0.29 - type: precision_at_1000 value: 0.083 - type: precision_at_3 value: 1.949 - type: precision_at_5 value: 1.567 - type: recall_at_1 value: 2.257 - type: recall_at_10 value: 7.382 - type: recall_at_100 value: 20.689 - type: recall_at_1000 value: 53.586 - type: recall_at_3 value: 4.786 - type: recall_at_5 value: 6.2829999999999995 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackPhysicsRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 6.691 - type: map_at_10 value: 9.447 - type: map_at_100 value: 10.174 - type: map_at_1000 value: 10.308 - type: map_at_3 value: 8.187999999999999 - type: map_at_5 value: 8.852 - type: mrr_at_1 value: 8.566 - type: mrr_at_10 value: 12.036 - type: mrr_at_100 value: 12.817 - type: mrr_at_1000 value: 12.918 - type: mrr_at_3 value: 10.539 - type: mrr_at_5 value: 11.381 - type: ndcg_at_1 value: 8.566 - type: ndcg_at_10 value: 11.95 - type: ndcg_at_100 value: 15.831000000000001 - type: ndcg_at_1000 value: 19.561 - type: ndcg_at_3 value: 9.467 - type: ndcg_at_5 value: 10.544 - type: precision_at_1 value: 8.566 - type: precision_at_10 value: 2.387 - type: precision_at_100 value: 0.538 - type: precision_at_1000 value: 0.104 - type: precision_at_3 value: 4.556 - type: precision_at_5 value: 3.5029999999999997 - type: recall_at_1 value: 6.691 - type: recall_at_10 value: 17.375 - type: recall_at_100 value: 34.503 - type: recall_at_1000 value: 61.492000000000004 - type: recall_at_3 value: 10.134 - type: recall_at_5 value: 13.056999999999999 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackProgrammersRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 4.68 - type: map_at_10 value: 6.776999999999999 - type: map_at_100 value: 7.207 - type: map_at_1000 value: 7.321999999999999 - type: map_at_3 value: 6.007 - type: map_at_5 value: 6.356000000000001 - type: mrr_at_1 value: 5.479 - type: mrr_at_10 value: 8.094999999999999 - type: mrr_at_100 value: 8.622 - type: mrr_at_1000 value: 8.729000000000001 - type: mrr_at_3 value: 7.249 - type: mrr_at_5 value: 7.6770000000000005 - type: ndcg_at_1 value: 5.479 - type: ndcg_at_10 value: 8.474 - type: ndcg_at_100 value: 11.134 - type: ndcg_at_1000 value: 14.759 - type: ndcg_at_3 value: 6.888 - type: ndcg_at_5 value: 7.504 - type: precision_at_1 value: 5.479 - type: precision_at_10 value: 1.575 - type: precision_at_100 value: 0.35000000000000003 - type: precision_at_1000 value: 0.08099999999999999 - type: precision_at_3 value: 3.272 - type: precision_at_5 value: 2.374 - type: recall_at_1 value: 4.68 - type: recall_at_10 value: 12.552 - type: recall_at_100 value: 24.91 - type: recall_at_1000 value: 52.019999999999996 - type: recall_at_3 value: 8.057 - type: recall_at_5 value: 9.629999999999999 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 4.741750000000001 - type: map_at_10 value: 7.103916666666667 - type: map_at_100 value: 7.656499999999998 - type: map_at_1000 value: 7.767583333333332 - type: map_at_3 value: 6.262416666666668 - type: map_at_5 value: 6.693916666666667 - type: mrr_at_1 value: 5.780583333333332 - type: mrr_at_10 value: 8.576333333333332 - type: mrr_at_100 value: 9.17975 - type: mrr_at_1000 value: 9.279083333333334 - type: mrr_at_3 value: 7.608833333333333 - type: mrr_at_5 value: 8.111333333333333 - type: ndcg_at_1 value: 5.780583333333332 - type: ndcg_at_10 value: 8.866166666666668 - type: ndcg_at_100 value: 12.037083333333333 - type: ndcg_at_1000 value: 15.4555 - type: ndcg_at_3 value: 7.179083333333335 - type: ndcg_at_5 value: 7.897166666666666 - type: precision_at_1 value: 5.780583333333332 - type: precision_at_10 value: 1.6935833333333334 - type: precision_at_100 value: 0.3921666666666667 - type: precision_at_1000 value: 0.08391666666666667 - type: precision_at_3 value: 3.425416666666666 - type: precision_at_5 value: 2.5570833333333334 - type: recall_at_1 value: 4.741750000000001 - type: recall_at_10 value: 12.889083333333334 - type: recall_at_100 value: 27.81866666666667 - type: recall_at_1000 value: 53.52316666666667 - type: recall_at_3 value: 8.179333333333332 - type: recall_at_5 value: 10.004083333333334 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackStatsRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 3.7130000000000005 - type: map_at_10 value: 5.734 - type: map_at_100 value: 6.297999999999999 - type: map_at_1000 value: 6.388000000000001 - type: map_at_3 value: 5.119 - type: map_at_5 value: 5.432 - type: mrr_at_1 value: 4.9079999999999995 - type: mrr_at_10 value: 7.2940000000000005 - type: mrr_at_100 value: 7.8549999999999995 - type: mrr_at_1000 value: 7.95 - type: mrr_at_3 value: 6.621 - type: mrr_at_5 value: 6.950000000000001 - type: ndcg_at_1 value: 4.9079999999999995 - type: ndcg_at_10 value: 7.167999999999999 - type: ndcg_at_100 value: 10.436 - type: ndcg_at_1000 value: 13.370999999999999 - type: ndcg_at_3 value: 5.959 - type: ndcg_at_5 value: 6.481000000000001 - type: precision_at_1 value: 4.9079999999999995 - type: precision_at_10 value: 1.3339999999999999 - type: precision_at_100 value: 0.33899999999999997 - type: precision_at_1000 value: 0.065 - type: precision_at_3 value: 2.965 - type: precision_at_5 value: 2.117 - type: recall_at_1 value: 3.7130000000000005 - type: recall_at_10 value: 10.156 - type: recall_at_100 value: 25.955000000000002 - type: recall_at_1000 value: 48.891 - type: recall_at_3 value: 6.795 - type: recall_at_5 value: 8.187999999999999 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackTexRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 2.114 - type: map_at_10 value: 3.4290000000000003 - type: map_at_100 value: 3.789 - type: map_at_1000 value: 3.878 - type: map_at_3 value: 2.9139999999999997 - type: map_at_5 value: 3.148 - type: mrr_at_1 value: 2.65 - type: mrr_at_10 value: 4.252000000000001 - type: mrr_at_100 value: 4.689 - type: mrr_at_1000 value: 4.782 - type: mrr_at_3 value: 3.671 - type: mrr_at_5 value: 3.9370000000000003 - type: ndcg_at_1 value: 2.65 - type: ndcg_at_10 value: 4.47 - type: ndcg_at_100 value: 6.654 - type: ndcg_at_1000 value: 9.713 - type: ndcg_at_3 value: 3.424 - type: ndcg_at_5 value: 3.794 - type: precision_at_1 value: 2.65 - type: precision_at_10 value: 0.9119999999999999 - type: precision_at_100 value: 0.248 - type: precision_at_1000 value: 0.063 - type: precision_at_3 value: 1.7209999999999999 - type: precision_at_5 value: 1.287 - type: recall_at_1 value: 2.114 - type: recall_at_10 value: 6.927 - type: recall_at_100 value: 17.26 - type: recall_at_1000 value: 40.672999999999995 - type: recall_at_3 value: 3.8859999999999997 - type: recall_at_5 value: 4.861 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackUnixRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 6.055 - type: map_at_10 value: 7.704999999999999 - type: map_at_100 value: 8.169 - type: map_at_1000 value: 8.257 - type: map_at_3 value: 7.033 - type: map_at_5 value: 7.4079999999999995 - type: mrr_at_1 value: 6.81 - type: mrr_at_10 value: 8.955 - type: mrr_at_100 value: 9.497 - type: mrr_at_1000 value: 9.583 - type: mrr_at_3 value: 8.116 - type: mrr_at_5 value: 8.526 - type: ndcg_at_1 value: 6.81 - type: ndcg_at_10 value: 9.113 - type: ndcg_at_100 value: 11.884 - type: ndcg_at_1000 value: 14.762 - type: ndcg_at_3 value: 7.675999999999999 - type: ndcg_at_5 value: 8.325000000000001 - type: precision_at_1 value: 6.81 - type: precision_at_10 value: 1.558 - type: precision_at_100 value: 0.34299999999999997 - type: precision_at_1000 value: 0.068 - type: precision_at_3 value: 3.2960000000000003 - type: precision_at_5 value: 2.388 - type: recall_at_1 value: 6.055 - type: recall_at_10 value: 12.219 - type: recall_at_100 value: 25.304 - type: recall_at_1000 value: 47.204 - type: recall_at_3 value: 8.387 - type: recall_at_5 value: 9.991 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWebmastersRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 5.043 - type: map_at_10 value: 7.394 - type: map_at_100 value: 8.096 - type: map_at_1000 value: 8.243 - type: map_at_3 value: 6.300999999999999 - type: map_at_5 value: 6.7780000000000005 - type: mrr_at_1 value: 6.126 - type: mrr_at_10 value: 9.308 - type: mrr_at_100 value: 10.091 - type: mrr_at_1000 value: 10.206 - type: mrr_at_3 value: 7.938000000000001 - type: mrr_at_5 value: 8.64 - type: ndcg_at_1 value: 6.126 - type: ndcg_at_10 value: 9.474 - type: ndcg_at_100 value: 13.238 - type: ndcg_at_1000 value: 17.366 - type: ndcg_at_3 value: 7.3260000000000005 - type: ndcg_at_5 value: 8.167 - type: precision_at_1 value: 6.126 - type: precision_at_10 value: 1.9959999999999998 - type: precision_at_100 value: 0.494 - type: precision_at_1000 value: 0.125 - type: precision_at_3 value: 3.557 - type: precision_at_5 value: 2.9250000000000003 - type: recall_at_1 value: 5.043 - type: recall_at_10 value: 13.812 - type: recall_at_100 value: 31.375999999999998 - type: recall_at_1000 value: 61.309999999999995 - type: recall_at_3 value: 7.8020000000000005 - type: recall_at_5 value: 9.725999999999999 - task: type: Retrieval dataset: type: BeIR/cqadupstack name: MTEB CQADupstackWordpressRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 3.771 - type: map_at_10 value: 5.152 - type: map_at_100 value: 5.584 - type: map_at_1000 value: 5.666 - type: map_at_3 value: 4.664 - type: map_at_5 value: 4.941 - type: mrr_at_1 value: 4.251 - type: mrr_at_10 value: 5.867 - type: mrr_at_100 value: 6.345000000000001 - type: mrr_at_1000 value: 6.432 - type: mrr_at_3 value: 5.36 - type: mrr_at_5 value: 5.656 - type: ndcg_at_1 value: 4.251 - type: ndcg_at_10 value: 6.16 - type: ndcg_at_100 value: 8.895 - type: ndcg_at_1000 value: 11.631 - type: ndcg_at_3 value: 5.176 - type: ndcg_at_5 value: 5.633 - type: precision_at_1 value: 4.251 - type: precision_at_10 value: 0.98 - type: precision_at_100 value: 0.259 - type: precision_at_1000 value: 0.053 - type: precision_at_3 value: 2.2800000000000002 - type: precision_at_5 value: 1.627 - type: recall_at_1 value: 3.771 - type: recall_at_10 value: 8.731 - type: recall_at_100 value: 22.517 - type: recall_at_1000 value: 44.183 - type: recall_at_3 value: 5.866 - type: recall_at_5 value: 7.066999999999999 - task: type: Retrieval dataset: type: climate-fever name: MTEB ClimateFEVER config: default split: test revision: None metrics: - type: map_at_1 value: 0.543 - type: map_at_10 value: 1.027 - type: map_at_100 value: 1.228 - type: map_at_1000 value: 1.266 - type: map_at_3 value: 0.756 - type: map_at_5 value: 0.877 - type: mrr_at_1 value: 1.3679999999999999 - type: mrr_at_10 value: 2.474 - type: mrr_at_100 value: 2.8369999999999997 - type: mrr_at_1000 value: 2.894 - type: mrr_at_3 value: 1.8780000000000001 - type: mrr_at_5 value: 2.1319999999999997 - type: ndcg_at_1 value: 1.3679999999999999 - type: ndcg_at_10 value: 1.791 - type: ndcg_at_100 value: 3.06 - type: ndcg_at_1000 value: 4.501 - type: ndcg_at_3 value: 1.16 - type: ndcg_at_5 value: 1.3419999999999999 - type: precision_at_1 value: 1.3679999999999999 - type: precision_at_10 value: 0.697 - type: precision_at_100 value: 0.193 - type: precision_at_1000 value: 0.045 - type: precision_at_3 value: 0.9339999999999999 - type: precision_at_5 value: 0.808 - type: recall_at_1 value: 0.543 - type: recall_at_10 value: 2.5149999999999997 - type: recall_at_100 value: 7.356999999999999 - type: recall_at_1000 value: 16.233 - type: recall_at_3 value: 1.018 - type: recall_at_5 value: 1.5150000000000001 - task: type: Retrieval dataset: type: C-MTEB/CmedqaRetrieval name: MTEB CmedqaRetrieval config: default split: dev revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301 metrics: - type: map_at_1 value: 3.7289999999999996 - type: map_at_10 value: 5.524 - type: map_at_100 value: 5.984 - type: map_at_1000 value: 6.087 - type: map_at_3 value: 4.854 - type: map_at_5 value: 5.2299999999999995 - type: mrr_at_1 value: 6.177 - type: mrr_at_10 value: 8.541 - type: mrr_at_100 value: 9.073 - type: mrr_at_1000 value: 9.161 - type: mrr_at_3 value: 7.71 - type: mrr_at_5 value: 8.148 - type: ndcg_at_1 value: 6.177 - type: ndcg_at_10 value: 7.217999999999999 - type: ndcg_at_100 value: 9.927 - type: ndcg_at_1000 value: 13.062000000000001 - type: ndcg_at_3 value: 6.0569999999999995 - type: ndcg_at_5 value: 6.544999999999999 - type: precision_at_1 value: 6.177 - type: precision_at_10 value: 1.6729999999999998 - type: precision_at_100 value: 0.38999999999999996 - type: precision_at_1000 value: 0.082 - type: precision_at_3 value: 3.5090000000000003 - type: precision_at_5 value: 2.596 - type: recall_at_1 value: 3.7289999999999996 - type: recall_at_10 value: 9.501 - type: recall_at_100 value: 21.444 - type: recall_at_1000 value: 43.891999999999996 - type: recall_at_3 value: 6.053 - type: recall_at_5 value: 7.531000000000001 - task: type: PairClassification dataset: type: C-MTEB/CMNLI name: MTEB Cmnli config: default split: validation revision: 41bc36f332156f7adc9e38f53777c959b2ae9766 metrics: - type: cos_sim_accuracy value: 58.123872519543 - type: cos_sim_ap value: 61.860457891021845 - type: cos_sim_f1 value: 68.18181818181817 - type: cos_sim_precision value: 52.4198617221873 - type: cos_sim_recall value: 97.49824643441664 - type: dot_accuracy value: 59.02585688514732 - type: dot_ap value: 62.10634470973564 - type: dot_f1 value: 68.3513630463003 - type: dot_precision value: 54.259411926353394 - type: dot_recall value: 92.33107318213702 - type: euclidean_accuracy value: 54.696331930246544 - type: euclidean_ap value: 59.58239795823632 - type: euclidean_f1 value: 67.95360660946935 - type: euclidean_precision value: 51.46191794007942 - type: euclidean_recall value: 100.0 - type: manhattan_accuracy value: 54.696331930246544 - type: manhattan_ap value: 59.593398063507394 - type: manhattan_f1 value: 67.94311591324383 - type: manhattan_precision value: 51.4560770156438 - type: manhattan_recall value: 99.9766191255553 - type: max_accuracy value: 59.02585688514732 - type: max_ap value: 62.10634470973564 - type: max_f1 value: 68.3513630463003 - task: type: Retrieval dataset: type: C-MTEB/CovidRetrieval name: MTEB CovidRetrieval config: default split: dev revision: 1271c7809071a13532e05f25fb53511ffce77117 metrics: - type: map_at_1 value: 6.428000000000001 - type: map_at_10 value: 8.883000000000001 - type: map_at_100 value: 9.549000000000001 - type: map_at_1000 value: 9.666 - type: map_at_3 value: 8.061 - type: map_at_5 value: 8.475000000000001 - type: mrr_at_1 value: 6.428000000000001 - type: mrr_at_10 value: 8.896999999999998 - type: mrr_at_100 value: 9.557 - type: mrr_at_1000 value: 9.674000000000001 - type: mrr_at_3 value: 8.061 - type: mrr_at_5 value: 8.488 - type: ndcg_at_1 value: 6.428000000000001 - type: ndcg_at_10 value: 10.382 - type: ndcg_at_100 value: 14.237 - type: ndcg_at_1000 value: 18.041 - type: ndcg_at_3 value: 8.613999999999999 - type: ndcg_at_5 value: 9.372 - type: precision_at_1 value: 6.428000000000001 - type: precision_at_10 value: 1.528 - type: precision_at_100 value: 0.349 - type: precision_at_1000 value: 0.067 - type: precision_at_3 value: 3.4070000000000005 - type: precision_at_5 value: 2.424 - type: recall_at_1 value: 6.428000000000001 - type: recall_at_10 value: 15.226999999999999 - type: recall_at_100 value: 34.694 - type: recall_at_1000 value: 66.07 - type: recall_at_3 value: 10.221 - type: recall_at_5 value: 12.065 - task: type: Retrieval dataset: type: dbpedia-entity name: MTEB DBPedia config: default split: test revision: None metrics: - type: map_at_1 value: 0.541 - type: map_at_10 value: 1.1560000000000001 - type: map_at_100 value: 1.508 - type: map_at_1000 value: 1.598 - type: map_at_3 value: 0.918 - type: map_at_5 value: 0.992 - type: mrr_at_1 value: 9.5 - type: mrr_at_10 value: 13.446 - type: mrr_at_100 value: 13.935 - type: mrr_at_1000 value: 14.008999999999999 - type: mrr_at_3 value: 12.083 - type: mrr_at_5 value: 12.733 - type: ndcg_at_1 value: 5.75 - type: ndcg_at_10 value: 3.9210000000000003 - type: ndcg_at_100 value: 3.975 - type: ndcg_at_1000 value: 5.634 - type: ndcg_at_3 value: 4.87 - type: ndcg_at_5 value: 4.259 - type: precision_at_1 value: 9.5 - type: precision_at_10 value: 3.9 - type: precision_at_100 value: 1.015 - type: precision_at_1000 value: 0.297 - type: precision_at_3 value: 6.75 - type: precision_at_5 value: 5.25 - type: recall_at_1 value: 0.541 - type: recall_at_10 value: 2.228 - type: recall_at_100 value: 4.9430000000000005 - type: recall_at_1000 value: 11.661000000000001 - type: recall_at_3 value: 1.264 - type: recall_at_5 value: 1.4869999999999999 - task: type: Classification dataset: type: DDSC/dkhate name: MTEB DKHateClassification config: default split: test revision: 59d12749a3c91a186063c7d729ec392fda94681c metrics: - type: accuracy value: 70.42553191489363 - type: ap value: 90.43423476644443 - type: f1 value: 55.82951993211343 - task: type: Classification dataset: type: AI-Sweden/SuperLim name: MTEB DalajClassification config: default split: test revision: 7ebf0b4caa7b2ae39698a889de782c09e6f5ee56 metrics: - type: accuracy value: 50.04504504504504 - type: ap value: 50.02574261238442 - type: f1 value: 49.48578760810561 - task: type: Classification dataset: type: danish_political_comments name: MTEB DanishPoliticalCommentsClassification config: default split: train revision: edbb03726c04a0efab14fc8c3b8b79e4d420e5a1 metrics: - type: accuracy value: 34.26450180405217 - type: f1 value: 30.14302584747605 - task: type: Retrieval dataset: type: C-MTEB/DuRetrieval name: MTEB DuRetrieval config: default split: dev revision: a1a333e290fe30b10f3f56498e3a0d911a693ced metrics: - type: map_at_1 value: 0.608 - type: map_at_10 value: 0.882 - type: map_at_100 value: 0.962 - type: map_at_1000 value: 1.028 - type: map_at_3 value: 0.749 - type: map_at_5 value: 0.8240000000000001 - type: mrr_at_1 value: 2.0500000000000003 - type: mrr_at_10 value: 2.796 - type: mrr_at_100 value: 2.983 - type: mrr_at_1000 value: 3.09 - type: mrr_at_3 value: 2.483 - type: mrr_at_5 value: 2.661 - type: ndcg_at_1 value: 2.0500000000000003 - type: ndcg_at_10 value: 1.435 - type: ndcg_at_100 value: 1.991 - type: ndcg_at_1000 value: 4.961 - type: ndcg_at_3 value: 1.428 - type: ndcg_at_5 value: 1.369 - type: precision_at_1 value: 2.0500000000000003 - type: precision_at_10 value: 0.5349999999999999 - type: precision_at_100 value: 0.127 - type: precision_at_1000 value: 0.086 - type: precision_at_3 value: 1.05 - type: precision_at_5 value: 0.84 - type: recall_at_1 value: 0.608 - type: recall_at_10 value: 1.54 - type: recall_at_100 value: 3.5069999999999997 - type: recall_at_1000 value: 20.531 - type: recall_at_3 value: 0.901 - type: recall_at_5 value: 1.168 - task: type: Retrieval dataset: type: C-MTEB/EcomRetrieval name: MTEB EcomRetrieval config: default split: dev revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9 metrics: - type: map_at_1 value: 3.1 - type: map_at_10 value: 4.016 - type: map_at_100 value: 4.455 - type: map_at_1000 value: 4.579 - type: map_at_3 value: 3.567 - type: map_at_5 value: 3.8019999999999996 - type: mrr_at_1 value: 3.1 - type: mrr_at_10 value: 4.016 - type: mrr_at_100 value: 4.455 - type: mrr_at_1000 value: 4.579 - type: mrr_at_3 value: 3.567 - type: mrr_at_5 value: 3.8019999999999996 - type: ndcg_at_1 value: 3.1 - type: ndcg_at_10 value: 4.684 - type: ndcg_at_100 value: 7.284 - type: ndcg_at_1000 value: 11.689 - type: ndcg_at_3 value: 3.7289999999999996 - type: ndcg_at_5 value: 4.146 - type: precision_at_1 value: 3.1 - type: precision_at_10 value: 0.69 - type: precision_at_100 value: 0.202 - type: precision_at_1000 value: 0.056999999999999995 - type: precision_at_3 value: 1.4000000000000001 - type: precision_at_5 value: 1.04 - type: recall_at_1 value: 3.1 - type: recall_at_10 value: 6.9 - type: recall_at_100 value: 20.200000000000003 - type: recall_at_1000 value: 57.3 - type: recall_at_3 value: 4.2 - type: recall_at_5 value: 5.2 - task: type: Classification dataset: type: mteb/emotion name: MTEB EmotionClassification config: default split: test revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 metrics: - type: accuracy value: 36.52 - type: f1 value: 33.66164203159101 - task: type: Retrieval dataset: type: fever name: MTEB FEVER config: default split: test revision: None metrics: - type: map_at_1 value: 0.9249999999999999 - type: map_at_10 value: 1.311 - type: map_at_100 value: 1.363 - type: map_at_1000 value: 1.376 - type: map_at_3 value: 1.145 - type: map_at_5 value: 1.233 - type: mrr_at_1 value: 0.975 - type: mrr_at_10 value: 1.371 - type: mrr_at_100 value: 1.426 - type: mrr_at_1000 value: 1.439 - type: mrr_at_3 value: 1.195 - type: mrr_at_5 value: 1.286 - type: ndcg_at_1 value: 0.975 - type: ndcg_at_10 value: 1.5859999999999999 - type: ndcg_at_100 value: 1.8800000000000001 - type: ndcg_at_1000 value: 2.313 - type: ndcg_at_3 value: 1.229 - type: ndcg_at_5 value: 1.388 - type: precision_at_1 value: 0.975 - type: precision_at_10 value: 0.254 - type: precision_at_100 value: 0.041 - type: precision_at_1000 value: 0.008 - type: precision_at_3 value: 0.49 - type: precision_at_5 value: 0.375 - type: recall_at_1 value: 0.9249999999999999 - type: recall_at_10 value: 2.4250000000000003 - type: recall_at_100 value: 3.866 - type: recall_at_1000 value: 7.401000000000001 - type: recall_at_3 value: 1.4200000000000002 - type: recall_at_5 value: 1.81 - task: type: Retrieval dataset: type: fiqa name: MTEB FiQA2018 config: default split: test revision: None metrics: - type: map_at_1 value: 1.032 - type: map_at_10 value: 1.8849999999999998 - type: map_at_100 value: 2.167 - type: map_at_1000 value: 2.266 - type: map_at_3 value: 1.609 - type: map_at_5 value: 1.7680000000000002 - type: mrr_at_1 value: 2.6229999999999998 - type: mrr_at_10 value: 4.479 - type: mrr_at_100 value: 4.92 - type: mrr_at_1000 value: 5.029999999999999 - type: mrr_at_3 value: 3.7289999999999996 - type: mrr_at_5 value: 4.138 - type: ndcg_at_1 value: 2.6229999999999998 - type: ndcg_at_10 value: 3.005 - type: ndcg_at_100 value: 5.01 - type: ndcg_at_1000 value: 8.312 - type: ndcg_at_3 value: 2.548 - type: ndcg_at_5 value: 2.735 - type: precision_at_1 value: 2.6229999999999998 - type: precision_at_10 value: 1.049 - type: precision_at_100 value: 0.31 - type: precision_at_1000 value: 0.089 - type: precision_at_3 value: 1.955 - type: precision_at_5 value: 1.574 - type: recall_at_1 value: 1.032 - type: recall_at_10 value: 3.888 - type: recall_at_100 value: 12.414 - type: recall_at_1000 value: 33.823 - type: recall_at_3 value: 2.37 - type: recall_at_5 value: 3.077 - task: type: Retrieval dataset: type: mteb/germanquad-retrieval name: MTEB GermanQuAD-Retrieval config: default split: test revision: f5c87ae5a2e7a5106606314eef45255f03151bb3 metrics: - type: map_at_1 value: 53.539 - type: map_at_10 value: 65.144 - type: map_at_100 value: 65.627 - type: map_at_1000 value: 65.63900000000001 - type: map_at_3 value: 62.598 - type: map_at_5 value: 64.302 - type: mrr_at_1 value: 53.539 - type: mrr_at_10 value: 65.144 - type: mrr_at_100 value: 65.627 - type: mrr_at_1000 value: 65.63900000000001 - type: mrr_at_3 value: 62.598 - type: mrr_at_5 value: 64.302 - type: ndcg_at_1 value: 53.539 - type: ndcg_at_10 value: 70.602 - type: ndcg_at_100 value: 72.886 - type: ndcg_at_1000 value: 73.14500000000001 - type: ndcg_at_3 value: 65.52900000000001 - type: ndcg_at_5 value: 68.596 - type: precision_at_1 value: 53.539 - type: precision_at_10 value: 8.757 - type: precision_at_100 value: 0.9809999999999999 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 24.667 - type: precision_at_5 value: 16.289 - type: recall_at_1 value: 53.539 - type: recall_at_10 value: 87.568 - type: recall_at_100 value: 98.09400000000001 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 74.002 - type: recall_at_5 value: 81.443 - task: type: Retrieval dataset: type: hotpotqa name: MTEB HotpotQA config: default split: test revision: None metrics: - type: map_at_1 value: 6.813 - type: map_at_10 value: 9.49 - type: map_at_100 value: 9.959 - type: map_at_1000 value: 10.024 - type: map_at_3 value: 8.618 - type: map_at_5 value: 9.084 - type: mrr_at_1 value: 13.626 - type: mrr_at_10 value: 17.818 - type: mrr_at_100 value: 18.412 - type: mrr_at_1000 value: 18.482000000000003 - type: mrr_at_3 value: 16.506999999999998 - type: mrr_at_5 value: 17.219 - type: ndcg_at_1 value: 13.626 - type: ndcg_at_10 value: 12.959999999999999 - type: ndcg_at_100 value: 15.562999999999999 - type: ndcg_at_1000 value: 17.571 - type: ndcg_at_3 value: 10.995000000000001 - type: ndcg_at_5 value: 11.908000000000001 - type: precision_at_1 value: 13.626 - type: precision_at_10 value: 2.995 - type: precision_at_100 value: 0.51 - type: precision_at_1000 value: 0.078 - type: precision_at_3 value: 7.000000000000001 - type: precision_at_5 value: 4.926 - type: recall_at_1 value: 6.813 - type: recall_at_10 value: 14.976 - type: recall_at_100 value: 25.517 - type: recall_at_1000 value: 39.095 - type: recall_at_3 value: 10.5 - type: recall_at_5 value: 12.316 - task: type: Classification dataset: type: C-MTEB/IFlyTek-classification name: MTEB IFlyTek config: default split: validation revision: 421605374b29664c5fc098418fe20ada9bd55f8a metrics: - type: accuracy value: 30.157752981916126 - type: f1 value: 18.801823560719093 - task: type: Classification dataset: type: mteb/imdb name: MTEB ImdbClassification config: default split: test revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 metrics: - type: accuracy value: 55.2292 - type: ap value: 53.00916532305935 - type: f1 value: 54.6906813069184 - task: type: Classification dataset: type: C-MTEB/JDReview-classification name: MTEB JDReview config: default split: test revision: b7c64bd89eb87f8ded463478346f76731f07bf8b metrics: - type: accuracy value: 66.73545966228893 - type: ap value: 26.119597600779542 - type: f1 value: 58.52011222432435 - task: type: STS dataset: type: C-MTEB/LCQMC name: MTEB LCQMC config: default split: test revision: 17f9b096f80380fce5ed12a9be8be7784b337daf metrics: - type: cos_sim_pearson value: 48.805858778250645 - type: cos_sim_spearman value: 50.1102532604773 - type: euclidean_pearson value: 48.94915568787808 - type: euclidean_spearman value: 49.20965979933496 - type: manhattan_pearson value: 48.913528193330855 - type: manhattan_spearman value: 49.151438504736994 - task: type: Classification dataset: type: DDSC/lcc name: MTEB LccSentimentClassification config: default split: test revision: de7ba3406ee55ea2cc52a0a41408fa6aede6d3c6 metrics: - type: accuracy value: 53.13333333333333 - type: f1 value: 51.09906324617105 - task: type: Reranking dataset: type: C-MTEB/Mmarco-reranking name: MTEB MMarcoReranking config: default split: dev revision: 8e0c766dbe9e16e1d221116a3f36795fbade07f6 metrics: - type: map value: 5.831060174627054 - type: mrr value: 4.019047619047618 - task: type: Retrieval dataset: type: C-MTEB/MMarcoRetrieval name: MTEB MMarcoRetrieval config: default split: dev revision: 539bbde593d947e2a124ba72651aafc09eb33fc2 metrics: - type: map_at_1 value: 5.826 - type: map_at_10 value: 8.956999999999999 - type: map_at_100 value: 9.746 - type: map_at_1000 value: 9.873999999999999 - type: map_at_3 value: 7.757 - type: map_at_5 value: 8.373 - type: mrr_at_1 value: 6.046 - type: mrr_at_10 value: 9.251 - type: mrr_at_100 value: 10.044 - type: mrr_at_1000 value: 10.167 - type: mrr_at_3 value: 8.028 - type: mrr_at_5 value: 8.66 - type: ndcg_at_1 value: 6.046 - type: ndcg_at_10 value: 10.998 - type: ndcg_at_100 value: 15.568999999999999 - type: ndcg_at_1000 value: 19.453 - type: ndcg_at_3 value: 8.468 - type: ndcg_at_5 value: 9.582 - type: precision_at_1 value: 6.046 - type: precision_at_10 value: 1.807 - type: precision_at_100 value: 0.42500000000000004 - type: precision_at_1000 value: 0.076 - type: precision_at_3 value: 3.572 - type: precision_at_5 value: 2.702 - type: recall_at_1 value: 5.826 - type: recall_at_10 value: 17.291 - type: recall_at_100 value: 40.037 - type: recall_at_1000 value: 71.351 - type: recall_at_3 value: 10.269 - type: recall_at_5 value: 12.950000000000001 - task: type: Retrieval dataset: type: msmarco name: MTEB MSMARCO config: default split: dev revision: None metrics: - type: map_at_1 value: 1.203 - type: map_at_10 value: 2.27 - type: map_at_100 value: 2.5860000000000003 - type: map_at_1000 value: 2.661 - type: map_at_3 value: 1.8159999999999998 - type: map_at_5 value: 2.037 - type: mrr_at_1 value: 1.232 - type: mrr_at_10 value: 2.338 - type: mrr_at_100 value: 2.665 - type: mrr_at_1000 value: 2.7390000000000003 - type: mrr_at_3 value: 1.87 - type: mrr_at_5 value: 2.1 - type: ndcg_at_1 value: 1.232 - type: ndcg_at_10 value: 3.005 - type: ndcg_at_100 value: 4.936 - type: ndcg_at_1000 value: 7.441000000000001 - type: ndcg_at_3 value: 2.036 - type: ndcg_at_5 value: 2.435 - type: precision_at_1 value: 1.232 - type: precision_at_10 value: 0.549 - type: precision_at_100 value: 0.158 - type: precision_at_1000 value: 0.038 - type: precision_at_3 value: 0.903 - type: precision_at_5 value: 0.739 - type: recall_at_1 value: 1.203 - type: recall_at_10 value: 5.332 - type: recall_at_100 value: 15.164 - type: recall_at_1000 value: 35.831 - type: recall_at_3 value: 2.622 - type: recall_at_5 value: 3.572 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (en) config: en split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 88.25809393524851 - type: f1 value: 87.67467702138386 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (de) config: de split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 87.86136939983095 - type: f1 value: 86.46545913246018 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (es) config: es split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 88.66911274182789 - type: f1 value: 87.56818480133035 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (fr) config: fr split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 82.32070153460695 - type: f1 value: 81.8684135195111 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (hi) config: hi split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 88.17497310864108 - type: f1 value: 87.77127061255791 - task: type: Classification dataset: type: mteb/mtop_domain name: MTEB MTOPDomainClassification (th) config: th split: test revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf metrics: - type: accuracy value: 83.61301989150091 - type: f1 value: 83.32707952781637 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (en) config: en split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 63.28545371637026 - type: f1 value: 39.28010634448113 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (de) config: de split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 67.51197520428289 - type: f1 value: 42.75299728819045 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (es) config: es split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 66.75783855903936 - type: f1 value: 41.9243384646481 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (fr) config: fr split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 57.61352959599124 - type: f1 value: 36.60494253557738 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (hi) config: hi split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 63.209035496593756 - type: f1 value: 38.34334001455857 - task: type: Classification dataset: type: mteb/mtop_intent name: MTEB MTOPIntentClassification (th) config: th split: test revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba metrics: - type: accuracy value: 62.01084990958409 - type: f1 value: 38.90143964367107 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (af) config: af split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 53.48016139878952 - type: f1 value: 51.413731172932984 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (am) config: am split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 52.74041694687289 - type: f1 value: 50.85469558689613 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ar) config: ar split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 52.42098184263618 - type: f1 value: 49.036138779692024 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (az) config: az split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 51.72158708809683 - type: f1 value: 50.92312837840828 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (bn) config: bn split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 57.71015467383995 - type: f1 value: 54.43623536118849 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (cy) config: cy split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 44.86886348352387 - type: f1 value: 45.25621242204162 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (da) config: da split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 55.554808338937455 - type: f1 value: 52.841205727784924 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (de) config: de split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 55.94149293880296 - type: f1 value: 52.93531236014481 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (el) config: el split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.95090786819098 - type: f1 value: 55.8924018128682 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (en) config: en split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 57.871553463349024 - type: f1 value: 55.48077228016125 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (es) config: es split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 59.05514458641561 - type: f1 value: 56.85428621465761 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fa) config: fa split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 62.51849361129792 - type: f1 value: 59.20555321128111 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fi) config: fi split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.13718897108272 - type: f1 value: 55.11327164614034 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (fr) config: fr split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 51.3046402151984 - type: f1 value: 49.54359751420772 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (he) config: he split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 59.512441156691324 - type: f1 value: 55.70948727453885 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hi) config: hi split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 55.69266980497647 - type: f1 value: 53.84873884279824 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hu) config: hu split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 57.821116341627445 - type: f1 value: 54.58226169327551 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (hy) config: hy split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 55.282447881640884 - type: f1 value: 52.46174777918203 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (id) config: id split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.49697377269669 - type: f1 value: 55.47458535278922 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (is) config: is split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 50.931405514458625 - type: f1 value: 48.76072294143928 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (it) config: it split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.920645595158035 - type: f1 value: 56.737389242501735 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ja) config: ja split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.890383322125096 - type: f1 value: 57.204112622120874 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (jv) config: jv split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 47.74714189643577 - type: f1 value: 45.39014412767061 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ka) config: ka split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 49.91930060524547 - type: f1 value: 48.95180513332781 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (km) config: km split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 37.989240080699396 - type: f1 value: 36.10638372264491 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (kn) config: kn split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 53.2985877605918 - type: f1 value: 51.12206506352444 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ko) config: ko split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.12373907195696 - type: f1 value: 56.204584512225274 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (lv) config: lv split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 50.11768661735038 - type: f1 value: 48.593654115832884 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ml) config: ml split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 57.91862811028917 - type: f1 value: 55.78809760336119 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (mn) config: mn split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 53.47679892400807 - type: f1 value: 50.59933850617468 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ms) config: ms split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 56.32145258910557 - type: f1 value: 53.78572864251967 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (my) config: my split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 53.544048419636844 - type: f1 value: 50.74399745452213 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nb) config: nb split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 55.05716207128446 - type: f1 value: 53.12820524166937 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (nl) config: nl split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 59.37794216543376 - type: f1 value: 56.614794144008386 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pl) config: pl split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 60.813718897108274 - type: f1 value: 57.15694453759643 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (pt) config: pt split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.399462004034966 - type: f1 value: 55.93483217710289 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ro) config: ro split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 57.763954270342964 - type: f1 value: 55.09638860069861 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ru) config: ru split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 55.65232010759921 - type: f1 value: 52.30181938086533 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sl) config: sl split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 58.74915938130465 - type: f1 value: 55.4593730920787 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sq) config: sq split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 56.90316072629456 - type: f1 value: 54.28504920273749 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sv) config: sv split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 57.52185608607936 - type: f1 value: 54.65171004538219 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (sw) config: sw split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 50.47074646940148 - type: f1 value: 49.91640094830554 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ta) config: ta split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 54.482178883658364 - type: f1 value: 52.24226771901219 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (te) config: te split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 54.64357767316746 - type: f1 value: 53.01487636439553 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (th) config: th split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 54.115669132481514 - type: f1 value: 52.413205142753206 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tl) config: tl split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 52.95225285810356 - type: f1 value: 50.75756649910168 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (tr) config: tr split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 51.66778749159382 - type: f1 value: 49.21139492760165 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (ur) config: ur split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 54.182918628110286 - type: f1 value: 52.039181194514875 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (vi) config: vi split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 55.59852051109617 - type: f1 value: 53.060686908268394 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-CN) config: zh-CN split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 57.726967047747145 - type: f1 value: 54.645702631584584 - task: type: Classification dataset: type: mteb/amazon_massive_intent name: MTEB MassiveIntentClassification (zh-TW) config: zh-TW split: test revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 metrics: - type: accuracy value: 51.42568930733018 - type: f1 value: 50.882328594696006 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (af) config: af split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 60.41694687289846 - type: f1 value: 58.690860679703725 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (am) config: am split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 59.48890383322125 - type: f1 value: 58.39835688515581 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ar) config: ar split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 60.238735709482185 - type: f1 value: 59.72662500042014 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (az) config: az split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 57.044384667115 - type: f1 value: 55.384095671817 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (bn) config: bn split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.35776731674514 - type: f1 value: 62.74311932650383 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (cy) config: cy split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 51.67787491593814 - type: f1 value: 50.38547848410452 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (da) config: da split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.98251513113653 - type: f1 value: 61.81962359822365 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (de) config: de split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 63.28177538668459 - type: f1 value: 62.03411861425649 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (el) config: el split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 66.29119031607263 - type: f1 value: 65.02800460526542 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (en) config: en split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.14593140551446 - type: f1 value: 62.99257031840126 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (es) config: es split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.43375924680565 - type: f1 value: 64.62403786298874 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fa) config: fa split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 68.90719569603229 - type: f1 value: 67.32949258170802 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fi) config: fi split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.00470746469402 - type: f1 value: 62.923780468763525 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (fr) config: fr split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 55.71284465366511 - type: f1 value: 56.08069685713033 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (he) config: he split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 66.3752521856086 - type: f1 value: 64.90995811394924 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hi) config: hi split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.858103564223256 - type: f1 value: 61.62638799458199 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hu) config: hu split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 66.2273032952253 - type: f1 value: 64.95618816406058 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (hy) config: hy split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.67652992602556 - type: f1 value: 61.33438873938238 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (id) config: id split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.45729657027573 - type: f1 value: 63.99738359920934 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (is) config: is split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 59.0719569603228 - type: f1 value: 57.518504167063234 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (it) config: it split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 66.13315400134499 - type: f1 value: 65.160633131251 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ja) config: ja split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 66.61398789509079 - type: f1 value: 65.79824135316741 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (jv) config: jv split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 56.90316072629456 - type: f1 value: 54.86322025311521 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ka) config: ka split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 56.94687289845326 - type: f1 value: 56.12621536279501 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (km) config: km split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 42.66308002689979 - type: f1 value: 38.86253489600552 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (kn) config: kn split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 61.20712844653665 - type: f1 value: 59.38664125148938 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ko) config: ko split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 66.16005379959651 - type: f1 value: 65.37496105527057 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (lv) config: lv split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 55.46738399462004 - type: f1 value: 54.74655536412335 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ml) config: ml split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.8675184936113 - type: f1 value: 64.4167690022513 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (mn) config: mn split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 59.34767989240082 - type: f1 value: 57.584089752919766 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ms) config: ms split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.92938802958977 - type: f1 value: 62.689467476962534 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (my) config: my split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 60.860793544048406 - type: f1 value: 59.642494854830964 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nb) config: nb split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 61.99058507061197 - type: f1 value: 60.80631394200557 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (nl) config: nl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.773369199731 - type: f1 value: 64.59673968371376 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pl) config: pl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 66.54673839946201 - type: f1 value: 65.39512803006458 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (pt) config: pt split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 63.64492266308002 - type: f1 value: 62.96480563325382 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ro) config: ro split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.03833221250841 - type: f1 value: 63.277354377232754 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ru) config: ru split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 63.09683927370544 - type: f1 value: 63.105668759138 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sl) config: sl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.635507733692 - type: f1 value: 64.1835797433392 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sq) config: sq split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.96973772696705 - type: f1 value: 63.000024177311595 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sv) config: sv split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.7848016139879 - type: f1 value: 63.36081856889688 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (sw) config: sw split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 59.97982515131136 - type: f1 value: 58.457979351949675 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ta) config: ta split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 61.19367854741089 - type: f1 value: 60.244347062989064 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (te) config: te split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.67989240080699 - type: f1 value: 61.03003364010584 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (th) config: th split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 64.70410221923335 - type: f1 value: 63.194634988237894 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tl) config: tl split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 60.45057162071284 - type: f1 value: 58.75729037108071 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (tr) config: tr split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 58.315400134498994 - type: f1 value: 59.46974195941193 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (ur) config: ur split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.562205783456626 - type: f1 value: 61.18682850669083 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (vi) config: vi split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 62.868190988567584 - type: f1 value: 62.04867947782604 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-CN) config: zh-CN split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 65.35642232683256 - type: f1 value: 63.940631681609915 - task: type: Classification dataset: type: mteb/amazon_massive_scenario name: MTEB MassiveScenarioClassification (zh-TW) config: zh-TW split: test revision: 7d571f92784cd94a019292a1f45445077d0ef634 metrics: - type: accuracy value: 58.81640887693342 - type: f1 value: 58.4125957354379 - task: type: Retrieval dataset: type: C-MTEB/MedicalRetrieval name: MTEB MedicalRetrieval config: default split: dev revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6 metrics: - type: map_at_1 value: 2.9000000000000004 - type: map_at_10 value: 3.5360000000000005 - type: map_at_100 value: 3.703 - type: map_at_1000 value: 3.734 - type: map_at_3 value: 3.167 - type: map_at_5 value: 3.322 - type: mrr_at_1 value: 2.9000000000000004 - type: mrr_at_10 value: 3.5360000000000005 - type: mrr_at_100 value: 3.703 - type: mrr_at_1000 value: 3.734 - type: mrr_at_3 value: 3.167 - type: mrr_at_5 value: 3.322 - type: ndcg_at_1 value: 2.9000000000000004 - type: ndcg_at_10 value: 4.079 - type: ndcg_at_100 value: 5.101 - type: ndcg_at_1000 value: 6.295000000000001 - type: ndcg_at_3 value: 3.276 - type: ndcg_at_5 value: 3.56 - type: precision_at_1 value: 2.9000000000000004 - type: precision_at_10 value: 0.59 - type: precision_at_100 value: 0.11199999999999999 - type: precision_at_1000 value: 0.022000000000000002 - type: precision_at_3 value: 1.2 - type: precision_at_5 value: 0.86 - type: recall_at_1 value: 2.9000000000000004 - type: recall_at_10 value: 5.8999999999999995 - type: recall_at_100 value: 11.200000000000001 - type: recall_at_1000 value: 21.5 - type: recall_at_3 value: 3.5999999999999996 - type: recall_at_5 value: 4.3 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-p2p name: MTEB MedrxivClusteringP2P config: default split: test revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 metrics: - type: v_measure value: 19.340284691485483 - task: type: Clustering dataset: type: mteb/medrxiv-clustering-s2s name: MTEB MedrxivClusteringS2S config: default split: test revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 metrics: - type: v_measure value: 17.003121027783283 - task: type: Reranking dataset: type: mteb/mind_small name: MTEB MindSmallReranking config: default split: test revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 metrics: - type: map value: 26.881341526634174 - type: mrr value: 27.23083862187728 - task: type: Classification dataset: type: C-MTEB/MultilingualSentiment-classification name: MTEB MultilingualSentiment config: default split: validation revision: 46958b007a63fdbf239b7672c25d0bea67b5ea1a metrics: - type: accuracy value: 54.08666666666667 - type: f1 value: 53.05912174796259 - task: type: Retrieval dataset: type: nfcorpus name: MTEB NFCorpus config: default split: test revision: None metrics: - type: map_at_1 value: 0.443 - type: map_at_10 value: 1.189 - type: map_at_100 value: 2.221 - type: map_at_1000 value: 3.034 - type: map_at_3 value: 0.683 - type: map_at_5 value: 0.882 - type: mrr_at_1 value: 4.334 - type: mrr_at_10 value: 10.908 - type: mrr_at_100 value: 12.536 - type: mrr_at_1000 value: 12.642000000000001 - type: mrr_at_3 value: 7.481999999999999 - type: mrr_at_5 value: 9.324 - type: ndcg_at_1 value: 3.7150000000000003 - type: ndcg_at_10 value: 5.591 - type: ndcg_at_100 value: 9.522 - type: ndcg_at_1000 value: 19.705000000000002 - type: ndcg_at_3 value: 4.292 - type: ndcg_at_5 value: 5.038 - type: precision_at_1 value: 4.334 - type: precision_at_10 value: 5.077 - type: precision_at_100 value: 3.2910000000000004 - type: precision_at_1000 value: 1.568 - type: precision_at_3 value: 4.644 - type: precision_at_5 value: 5.139 - type: recall_at_1 value: 0.443 - type: recall_at_10 value: 3.3520000000000003 - type: recall_at_100 value: 15.515 - type: recall_at_1000 value: 50.505 - type: recall_at_3 value: 0.931 - type: recall_at_5 value: 1.698 - task: type: Retrieval dataset: type: nq name: MTEB NQ config: default split: test revision: None metrics: - type: map_at_1 value: 0.439 - type: map_at_10 value: 0.6839999999999999 - type: map_at_100 value: 0.769 - type: map_at_1000 value: 0.79 - type: map_at_3 value: 0.584 - type: map_at_5 value: 0.621 - type: mrr_at_1 value: 0.5499999999999999 - type: mrr_at_10 value: 0.819 - type: mrr_at_100 value: 0.9169999999999999 - type: mrr_at_1000 value: 0.9400000000000001 - type: mrr_at_3 value: 0.705 - type: mrr_at_5 value: 0.75 - type: ndcg_at_1 value: 0.5499999999999999 - type: ndcg_at_10 value: 0.886 - type: ndcg_at_100 value: 1.422 - type: ndcg_at_1000 value: 2.2079999999999997 - type: ndcg_at_3 value: 0.6629999999999999 - type: ndcg_at_5 value: 0.735 - type: precision_at_1 value: 0.5499999999999999 - type: precision_at_10 value: 0.16199999999999998 - type: precision_at_100 value: 0.048 - type: precision_at_1000 value: 0.012 - type: precision_at_3 value: 0.309 - type: precision_at_5 value: 0.22599999999999998 - type: recall_at_1 value: 0.439 - type: recall_at_10 value: 1.405 - type: recall_at_100 value: 4.051 - type: recall_at_1000 value: 10.487 - type: recall_at_3 value: 0.787 - type: recall_at_5 value: 0.9560000000000001 - task: type: Retrieval dataset: type: narrativeqa name: MTEB NarrativeQARetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 5.93 - type: map_at_10 value: 7.349 - type: map_at_100 value: 8.011 - type: map_at_1000 value: 8.351 - type: map_at_3 value: 6.787 - type: map_at_5 value: 7.02 - type: mrr_at_1 value: 5.93 - type: mrr_at_10 value: 7.349 - type: mrr_at_100 value: 8.011 - type: mrr_at_1000 value: 8.351 - type: mrr_at_3 value: 6.787 - type: mrr_at_5 value: 7.02 - type: ndcg_at_1 value: 5.93 - type: ndcg_at_10 value: 8.291 - type: ndcg_at_100 value: 12.833 - type: ndcg_at_1000 value: 21.253 - type: ndcg_at_3 value: 7.072000000000001 - type: ndcg_at_5 value: 7.495 - type: precision_at_1 value: 5.93 - type: precision_at_10 value: 1.1400000000000001 - type: precision_at_100 value: 0.359 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 2.633 - type: precision_at_5 value: 1.786 - type: recall_at_1 value: 5.93 - type: recall_at_10 value: 11.395 - type: recall_at_100 value: 35.929 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 7.9 - type: recall_at_5 value: 8.932 - task: type: Classification dataset: type: ScandEval/norec-mini name: MTEB NoRecClassification config: default split: test revision: 07b99ab3363c2e7f8f87015b01c21f4d9b917ce3 metrics: - type: accuracy value: 48.720703125 - type: f1 value: 44.40194414610097 - task: type: Classification dataset: type: strombergnlp/nordic_langid name: MTEB NordicLangClassification config: default split: test revision: e254179d18ab0165fdb6dbef91178266222bee2a metrics: - type: accuracy value: 46.626666666666665 - type: f1 value: 44.771861545457476 - task: type: BitextMining dataset: type: kardosdrur/norwegian-courts name: MTEB NorwegianCourtsBitextMining config: default split: test revision: None metrics: - type: accuracy value: 94.73684210526315 - type: f1 value: 93.85964912280701 - type: precision value: 93.42105263157895 - type: recall value: 94.73684210526315 - task: type: Classification dataset: type: NbAiLab/norwegian_parliament name: MTEB NorwegianParliament config: default split: test revision: f7393532774c66312378d30b197610b43d751972 metrics: - type: accuracy value: 55.408333333333324 - type: ap value: 53.03584049725229 - type: f1 value: 55.0030558323002 - task: type: PairClassification dataset: type: C-MTEB/OCNLI name: MTEB Ocnli config: default split: validation revision: 66e76a618a34d6d565d5538088562851e6daa7ec metrics: - type: cos_sim_accuracy value: 54.68327016783974 - type: cos_sim_ap value: 55.175113037501625 - type: cos_sim_f1 value: 67.81733189500179 - type: cos_sim_precision value: 51.41766630316249 - type: cos_sim_recall value: 99.57761351636748 - type: dot_accuracy value: 53.3838657282079 - type: dot_ap value: 54.09970939118489 - type: dot_f1 value: 67.81485468245427 - type: dot_precision value: 51.358695652173914 - type: dot_recall value: 99.78880675818374 - type: euclidean_accuracy value: 52.680021656740664 - type: euclidean_ap value: 54.61964760120384 - type: euclidean_f1 value: 67.86743515850145 - type: euclidean_precision value: 51.50355385456533 - type: euclidean_recall value: 99.47201689545935 - type: manhattan_accuracy value: 52.84244721169464 - type: manhattan_ap value: 54.95963401318357 - type: manhattan_f1 value: 67.96818510484454 - type: manhattan_precision value: 51.67674546454095 - type: manhattan_recall value: 99.26082365364309 - type: max_accuracy value: 54.68327016783974 - type: max_ap value: 55.175113037501625 - type: max_f1 value: 67.96818510484454 - task: type: Classification dataset: type: C-MTEB/OnlineShopping-classification name: MTEB OnlineShopping config: default split: test revision: e610f2ebd179a8fda30ae534c3878750a96db120 metrics: - type: accuracy value: 71.38 - type: ap value: 67.98361406325826 - type: f1 value: 71.32677133367216 - task: type: Classification dataset: type: laugustyniak/abusive-clauses-pl name: MTEB PAC config: default split: test revision: None metrics: - type: accuracy value: 73.79090645815232 - type: ap value: 79.78709287447715 - type: f1 value: 71.11540289567428 - task: type: STS dataset: type: C-MTEB/PAWSX name: MTEB PAWSX config: default split: test revision: 9c6a90e430ac22b5779fb019a23e820b11a8b5e1 metrics: - type: cos_sim_pearson value: 29.52361484634536 - type: cos_sim_spearman value: 32.749971481829185 - type: euclidean_pearson value: 32.54178605588255 - type: euclidean_spearman value: 31.470626091963805 - type: manhattan_pearson value: 32.59570732501661 - type: manhattan_spearman value: 31.45843300214921 - task: type: PairClassification dataset: type: PL-MTEB/ppc-pairclassification name: MTEB PPC config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 71.1 - type: cos_sim_ap value: 85.36544548691205 - type: cos_sim_f1 value: 75.23393636930756 - type: cos_sim_precision value: 60.36036036036037 - type: cos_sim_recall value: 99.83443708609272 - type: dot_accuracy value: 64.1 - type: dot_ap value: 68.24516982673607 - type: dot_f1 value: 75.9412890874282 - type: dot_precision value: 61.78608515057113 - type: dot_recall value: 98.50993377483444 - type: euclidean_accuracy value: 73.0 - type: euclidean_ap value: 85.77605840690839 - type: euclidean_f1 value: 75.23393636930756 - type: euclidean_precision value: 60.36036036036037 - type: euclidean_recall value: 99.83443708609272 - type: manhattan_accuracy value: 73.0 - type: manhattan_ap value: 85.77723100027544 - type: manhattan_f1 value: 75.23393636930756 - type: manhattan_precision value: 60.36036036036037 - type: manhattan_recall value: 99.83443708609272 - type: max_accuracy value: 73.0 - type: max_ap value: 85.77723100027544 - type: max_f1 value: 75.9412890874282 - task: type: PairClassification dataset: type: PL-MTEB/psc-pairclassification name: MTEB PSC config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 90.81632653061224 - type: cos_sim_ap value: 91.97693749083473 - type: cos_sim_f1 value: 85.55078683834049 - type: cos_sim_precision value: 80.59299191374663 - type: cos_sim_recall value: 91.15853658536585 - type: dot_accuracy value: 85.9925788497217 - type: dot_ap value: 83.51557404204843 - type: dot_f1 value: 78.0141843971631 - type: dot_precision value: 72.94429708222812 - type: dot_recall value: 83.84146341463415 - type: euclidean_accuracy value: 88.12615955473099 - type: euclidean_ap value: 88.83596272873487 - type: euclidean_f1 value: 80.46647230320698 - type: euclidean_precision value: 77.09497206703911 - type: euclidean_recall value: 84.14634146341463 - type: manhattan_accuracy value: 88.31168831168831 - type: manhattan_ap value: 89.15769755969164 - type: manhattan_f1 value: 80.62678062678062 - type: manhattan_precision value: 75.66844919786097 - type: manhattan_recall value: 86.28048780487805 - type: max_accuracy value: 90.81632653061224 - type: max_ap value: 91.97693749083473 - type: max_f1 value: 85.55078683834049 - task: type: Classification dataset: type: PL-MTEB/polemo2_in name: MTEB PolEmo2.0-IN config: default split: test revision: None metrics: - type: accuracy value: 50.72022160664821 - type: f1 value: 51.12376375708585 - task: type: Classification dataset: type: PL-MTEB/polemo2_out name: MTEB PolEmo2.0-OUT config: default split: test revision: None metrics: - type: accuracy value: 33.36032388663968 - type: f1 value: 27.16896572816315 - task: type: STS dataset: type: C-MTEB/QBQTC name: MTEB QBQTC config: default split: test revision: 790b0510dc52b1553e8c49f3d2afb48c0e5c48b7 metrics: - type: cos_sim_pearson value: 23.34232603207876 - type: cos_sim_spearman value: 24.4795842135422 - type: euclidean_pearson value: 19.94484261154701 - type: euclidean_spearman value: 20.678133816020438 - type: manhattan_pearson value: 22.39245233528255 - type: manhattan_spearman value: 22.93218231516663 - task: type: Retrieval dataset: type: quora name: MTEB QuoraRetrieval config: default split: test revision: None metrics: - type: map_at_1 value: 61.451 - type: map_at_10 value: 73.898 - type: map_at_100 value: 74.72800000000001 - type: map_at_1000 value: 74.766 - type: map_at_3 value: 70.994 - type: map_at_5 value: 72.74000000000001 - type: mrr_at_1 value: 70.94 - type: mrr_at_10 value: 78.35600000000001 - type: mrr_at_100 value: 78.64 - type: mrr_at_1000 value: 78.647 - type: mrr_at_3 value: 76.912 - type: mrr_at_5 value: 77.81 - type: ndcg_at_1 value: 70.93 - type: ndcg_at_10 value: 78.622 - type: ndcg_at_100 value: 81.00699999999999 - type: ndcg_at_1000 value: 81.453 - type: ndcg_at_3 value: 75.03999999999999 - type: ndcg_at_5 value: 76.839 - type: precision_at_1 value: 70.93 - type: precision_at_10 value: 11.953 - type: precision_at_100 value: 1.4489999999999998 - type: precision_at_1000 value: 0.154 - type: precision_at_3 value: 32.65 - type: precision_at_5 value: 21.598 - type: recall_at_1 value: 61.451 - type: recall_at_10 value: 87.608 - type: recall_at_100 value: 96.818 - type: recall_at_1000 value: 99.445 - type: recall_at_3 value: 77.348 - type: recall_at_5 value: 82.334 - task: type: Clustering dataset: type: mteb/reddit-clustering name: MTEB RedditClustering config: default split: test revision: 24640382cdbf8abc73003fb0fa6d111a705499eb metrics: - type: v_measure value: 18.928966218247353 - task: type: Clustering dataset: type: mteb/reddit-clustering-p2p name: MTEB RedditClusteringP2P config: default split: test revision: 282350215ef01743dc01b456c7f5241fa8937f16 metrics: - type: v_measure value: 35.948929675640414 - task: type: Retrieval dataset: type: scidocs name: MTEB SCIDOCS config: default split: test revision: None metrics: - type: map_at_1 value: 0.52 - type: map_at_10 value: 0.893 - type: map_at_100 value: 1.113 - type: map_at_1000 value: 1.304 - type: map_at_3 value: 0.7779999999999999 - type: map_at_5 value: 0.8200000000000001 - type: mrr_at_1 value: 2.6 - type: mrr_at_10 value: 4.0680000000000005 - type: mrr_at_100 value: 4.6080000000000005 - type: mrr_at_1000 value: 4.797 - type: mrr_at_3 value: 3.5999999999999996 - type: mrr_at_5 value: 3.8150000000000004 - type: ndcg_at_1 value: 2.6 - type: ndcg_at_10 value: 1.79 - type: ndcg_at_100 value: 3.5549999999999997 - type: ndcg_at_1000 value: 9.942 - type: ndcg_at_3 value: 1.94 - type: ndcg_at_5 value: 1.543 - type: precision_at_1 value: 2.6 - type: precision_at_10 value: 0.8500000000000001 - type: precision_at_100 value: 0.361 - type: precision_at_1000 value: 0.197 - type: precision_at_3 value: 1.7670000000000001 - type: precision_at_5 value: 1.26 - type: recall_at_1 value: 0.52 - type: recall_at_10 value: 1.7149999999999999 - type: recall_at_100 value: 7.318 - type: recall_at_1000 value: 39.915 - type: recall_at_3 value: 1.0699999999999998 - type: recall_at_5 value: 1.27 - task: type: PairClassification dataset: type: PL-MTEB/sicke-pl-pairclassification name: MTEB SICK-E-PL config: default split: test revision: None metrics: - type: cos_sim_accuracy value: 73.84834896045659 - type: cos_sim_ap value: 55.48406649230719 - type: cos_sim_f1 value: 57.34228187919464 - type: cos_sim_precision value: 46.01464885825076 - type: cos_sim_recall value: 76.06837606837607 - type: dot_accuracy value: 72.6049735018345 - type: dot_ap value: 47.4643427557602 - type: dot_f1 value: 55.22423104015027 - type: dot_precision value: 41.190893169877405 - type: dot_recall value: 83.76068376068376 - type: euclidean_accuracy value: 74.01141459437423 - type: euclidean_ap value: 55.32291633102242 - type: euclidean_f1 value: 56.970649895178184 - type: euclidean_precision value: 45.066334991708125 - type: euclidean_recall value: 77.42165242165242 - type: manhattan_accuracy value: 73.99103139013454 - type: manhattan_ap value: 55.25996778114356 - type: manhattan_f1 value: 57.0203644158628 - type: manhattan_precision value: 45.70446735395189 - type: manhattan_recall value: 75.78347578347578 - type: max_accuracy value: 74.01141459437423 - type: max_ap value: 55.48406649230719 - type: max_f1 value: 57.34228187919464 - task: type: STS dataset: type: mteb/sickr-sts name: MTEB SICK-R config: default split: test revision: a6ea5a8cab320b040a23452cc28066d9beae2cee metrics: - type: cos_sim_pearson value: 67.0394316443087 - type: cos_sim_spearman value: 62.93973839554845 - type: euclidean_pearson value: 57.60094512966203 - type: euclidean_spearman value: 57.05926985559664 - type: manhattan_pearson value: 57.53054316732484 - type: manhattan_spearman value: 56.98810707222501 - task: type: STS dataset: type: PL-MTEB/sickr-pl-sts name: MTEB SICK-R-PL config: default split: test revision: None metrics: - type: cos_sim_pearson value: 67.75034195626081 - type: cos_sim_spearman value: 62.98156510096241 - type: euclidean_pearson value: 60.102373885669 - type: euclidean_spearman value: 58.78567620580185 - type: manhattan_pearson value: 60.06786982256782 - type: manhattan_spearman value: 58.782787914089305 - task: type: STS dataset: type: mteb/sts12-sts name: MTEB STS12 config: default split: test revision: a0d554a64d88156834ff5ae9920b964011b16384 metrics: - type: cos_sim_pearson value: 61.62220407968201 - type: cos_sim_spearman value: 65.46331444956428 - type: euclidean_pearson value: 53.17721859223576 - type: euclidean_spearman value: 62.26490113361859 - type: manhattan_pearson value: 53.24730859766875 - type: manhattan_spearman value: 62.27695125302589 - task: type: STS dataset: type: mteb/sts13-sts name: MTEB STS13 config: default split: test revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca metrics: - type: cos_sim_pearson value: 60.32974355127381 - type: cos_sim_spearman value: 62.786126823027246 - type: euclidean_pearson value: 48.85507668127239 - type: euclidean_spearman value: 50.87786603451179 - type: manhattan_pearson value: 49.12859466063693 - type: manhattan_spearman value: 50.84858327663952 - task: type: STS dataset: type: mteb/sts14-sts name: MTEB STS14 config: default split: test revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 metrics: - type: cos_sim_pearson value: 59.1627744693388 - type: cos_sim_spearman value: 57.53692924942505 - type: euclidean_pearson value: 43.86657039278782 - type: euclidean_spearman value: 45.082440020418424 - type: manhattan_pearson value: 43.955740479348044 - type: manhattan_spearman value: 45.15905727160372 - task: type: STS dataset: type: mteb/sts15-sts name: MTEB STS15 config: default split: test revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 metrics: - type: cos_sim_pearson value: 74.42588616006334 - type: cos_sim_spearman value: 74.25087851075787 - type: euclidean_pearson value: 63.646746727259774 - type: euclidean_spearman value: 63.150219043712106 - type: manhattan_pearson value: 63.5341614083317 - type: manhattan_spearman value: 63.00883784164611 - task: type: STS dataset: type: mteb/sts16-sts name: MTEB STS16 config: default split: test revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 metrics: - type: cos_sim_pearson value: 75.77242395196524 - type: cos_sim_spearman value: 75.72930700521489 - type: euclidean_pearson value: 68.76523039736611 - type: euclidean_spearman value: 67.96962494828459 - type: manhattan_pearson value: 68.66742487680166 - type: manhattan_spearman value: 67.89168383123581 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ko-ko) config: ko-ko split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 66.24952916970275 - type: cos_sim_spearman value: 66.14121993606645 - type: euclidean_pearson value: 48.49193377121151 - type: euclidean_spearman value: 48.30443629726529 - type: manhattan_pearson value: 48.580679507100214 - type: manhattan_spearman value: 48.163570736145374 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (ar-ar) config: ar-ar split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 75.00451698478369 - type: cos_sim_spearman value: 74.96504390762252 - type: euclidean_pearson value: 71.41544320562012 - type: euclidean_spearman value: 69.8671968302191 - type: manhattan_pearson value: 71.51396887529955 - type: manhattan_spearman value: 70.13646740313014 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-ar) config: en-ar split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 73.88477970895238 - type: cos_sim_spearman value: 74.05322141272103 - type: euclidean_pearson value: 60.60761818278518 - type: euclidean_spearman value: 59.33251110095505 - type: manhattan_pearson value: 60.419868469543594 - type: manhattan_spearman value: 59.298607116416505 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-de) config: en-de split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 75.57014739802696 - type: cos_sim_spearman value: 74.95329129562734 - type: euclidean_pearson value: 66.04424909946401 - type: euclidean_spearman value: 65.36290486357399 - type: manhattan_pearson value: 65.85564903072199 - type: manhattan_spearman value: 65.1132393036515 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-en) config: en-en split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 80.04007186284058 - type: cos_sim_spearman value: 79.94429611477106 - type: euclidean_pearson value: 73.04121359782312 - type: euclidean_spearman value: 72.46378547225223 - type: manhattan_pearson value: 73.20754563807353 - type: manhattan_spearman value: 72.61008679959097 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (en-tr) config: en-tr split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 77.28740927858394 - type: cos_sim_spearman value: 77.18286391819586 - type: euclidean_pearson value: 67.75873608167655 - type: euclidean_spearman value: 66.81787653005944 - type: manhattan_pearson value: 67.77279216343489 - type: manhattan_spearman value: 66.9579776982823 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-en) config: es-en split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 72.99292955338515 - type: cos_sim_spearman value: 72.24657859872468 - type: euclidean_pearson value: 63.52202589317903 - type: euclidean_spearman value: 62.33443239002958 - type: manhattan_pearson value: 63.769212052975476 - type: manhattan_spearman value: 62.81347262019862 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (es-es) config: es-es split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 81.37213847952495 - type: cos_sim_spearman value: 80.64548512701263 - type: euclidean_pearson value: 78.20509452005497 - type: euclidean_spearman value: 76.92821412749807 - type: manhattan_pearson value: 78.18187623856416 - type: manhattan_spearman value: 76.80662828217807 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (fr-en) config: fr-en split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 77.54059838849403 - type: cos_sim_spearman value: 77.19559169700682 - type: euclidean_pearson value: 65.48086288727644 - type: euclidean_spearman value: 65.00387848487095 - type: manhattan_pearson value: 65.39875192421043 - type: manhattan_spearman value: 64.99073212282424 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (it-en) config: it-en split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 79.06396980015494 - type: cos_sim_spearman value: 78.66694637555244 - type: euclidean_pearson value: 69.77559635500967 - type: euclidean_spearman value: 69.6548076752796 - type: manhattan_pearson value: 69.80486003899448 - type: manhattan_spearman value: 69.64277606323098 - task: type: STS dataset: type: mteb/sts17-crosslingual-sts name: MTEB STS17 (nl-en) config: nl-en split: test revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d metrics: - type: cos_sim_pearson value: 78.66045887460483 - type: cos_sim_spearman value: 78.14055373851183 - type: euclidean_pearson value: 69.24649434854432 - type: euclidean_spearman value: 68.92768315926817 - type: manhattan_pearson value: 69.53186552758453 - type: manhattan_spearman value: 69.1888421760678 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (en) config: en split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 36.63454695072987 - type: cos_sim_spearman value: 47.12016162570126 - type: euclidean_pearson value: 40.706330494343604 - type: euclidean_spearman value: 48.865006150948616 - type: manhattan_pearson value: 45.099280138151926 - type: manhattan_spearman value: 51.15098931364965 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de) config: de split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 13.32554927123781 - type: cos_sim_spearman value: 24.094051740693416 - type: euclidean_pearson value: 5.733247674182589 - type: euclidean_spearman value: 17.32765054587116 - type: manhattan_pearson value: 7.85068670554767 - type: manhattan_spearman value: 19.003823576388488 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es) config: es split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 41.212816807176836 - type: cos_sim_spearman value: 55.397885077207974 - type: euclidean_pearson value: 44.32687000784094 - type: euclidean_spearman value: 54.40126826356849 - type: manhattan_pearson value: 49.72970710546519 - type: manhattan_spearman value: 56.49113106101981 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl) config: pl split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 5.985016492310366 - type: cos_sim_spearman value: 25.307464943919012 - type: euclidean_pearson value: -6.568128940119214 - type: euclidean_spearman value: 23.37422025614559 - type: manhattan_pearson value: -4.99967062376341 - type: manhattan_spearman value: 25.00987603568494 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (tr) config: tr split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 34.44666733247214 - type: cos_sim_spearman value: 46.45977141800899 - type: euclidean_pearson value: 38.168335484005496 - type: euclidean_spearman value: 45.87203585565584 - type: manhattan_pearson value: 45.96851222581045 - type: manhattan_spearman value: 50.72243559635815 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ar) config: ar split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 28.095330260570044 - type: cos_sim_spearman value: 40.24741745875892 - type: euclidean_pearson value: 34.298302094447 - type: euclidean_spearman value: 41.75099510117058 - type: manhattan_pearson value: 36.822574344860016 - type: manhattan_spearman value: 43.24173881526762 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (ru) config: ru split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 27.467875570822507 - type: cos_sim_spearman value: 43.57385391855994 - type: euclidean_pearson value: 22.17414485325322 - type: euclidean_spearman value: 39.95656416250951 - type: manhattan_pearson value: 26.066273487373664 - type: manhattan_spearman value: 40.14230396654245 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh) config: zh split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 34.87842153404196 - type: cos_sim_spearman value: 52.8221686505807 - type: euclidean_pearson value: 35.93221146007856 - type: euclidean_spearman value: 51.489361135954105 - type: manhattan_pearson value: 44.030913421863026 - type: manhattan_spearman value: 52.999521693608465 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr) config: fr split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 49.999270334861315 - type: cos_sim_spearman value: 61.71837669697145 - type: euclidean_pearson value: 49.955296540972824 - type: euclidean_spearman value: 56.505586159244515 - type: manhattan_pearson value: 52.49957702118715 - type: manhattan_spearman value: 57.12582682664643 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-en) config: de-en split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 22.43564766265908 - type: cos_sim_spearman value: 34.28346144104183 - type: euclidean_pearson value: 33.017091607053786 - type: euclidean_spearman value: 37.60239300835095 - type: manhattan_pearson value: 43.842762856035904 - type: manhattan_spearman value: 44.71536370367842 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-en) config: es-en split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 44.34071373662986 - type: cos_sim_spearman value: 57.823185616169646 - type: euclidean_pearson value: 53.672221309748835 - type: euclidean_spearman value: 62.001696407713126 - type: manhattan_pearson value: 61.30617811516088 - type: manhattan_spearman value: 63.97035045686644 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (it) config: it split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 56.07027264854201 - type: cos_sim_spearman value: 62.20158260763411 - type: euclidean_pearson value: 55.89068916951512 - type: euclidean_spearman value: 61.46687669950822 - type: manhattan_pearson value: 55.82045134105482 - type: manhattan_spearman value: 61.097021240849394 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (pl-en) config: pl-en split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 41.418158049627344 - type: cos_sim_spearman value: 44.71684955996943 - type: euclidean_pearson value: 44.09873868592676 - type: euclidean_spearman value: 48.582169034980765 - type: manhattan_pearson value: 51.11307812290116 - type: manhattan_spearman value: 51.613996439908796 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (zh-en) config: zh-en split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 38.89141661170763 - type: cos_sim_spearman value: 49.18633779347391 - type: euclidean_pearson value: 48.37473555026282 - type: euclidean_spearman value: 54.17536451762477 - type: manhattan_pearson value: 55.06464763414177 - type: manhattan_spearman value: 55.27394644455171 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (es-it) config: es-it split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 40.31621370101309 - type: cos_sim_spearman value: 49.13034440774138 - type: euclidean_pearson value: 46.55326804225437 - type: euclidean_spearman value: 50.04387120070582 - type: manhattan_pearson value: 51.108871970953615 - type: manhattan_spearman value: 50.77822527742335 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-fr) config: de-fr split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 27.07459941263371 - type: cos_sim_spearman value: 41.29498619968451 - type: euclidean_pearson value: 28.81148557875536 - type: euclidean_spearman value: 44.23270186092313 - type: manhattan_pearson value: 33.37331199208485 - type: manhattan_spearman value: 44.702014347511174 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (de-pl) config: de-pl split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 6.864350141604963 - type: cos_sim_spearman value: 25.805458732687914 - type: euclidean_pearson value: 22.41182782328486 - type: euclidean_spearman value: 41.47965741399835 - type: manhattan_pearson value: 27.52235896097151 - type: manhattan_spearman value: 45.11336162695441 - task: type: STS dataset: type: mteb/sts22-crosslingual-sts name: MTEB STS22 (fr-pl) config: fr-pl split: test revision: eea2b4fe26a775864c896887d910b76a8098ad3f metrics: - type: cos_sim_pearson value: 65.4403182219975 - type: cos_sim_spearman value: 61.97797868009122 - type: euclidean_pearson value: 57.9607330049556 - type: euclidean_spearman value: 50.709255283710995 - type: manhattan_pearson value: 61.3956353319538 - type: manhattan_spearman value: 61.97797868009122 - task: type: STS dataset: type: C-MTEB/STSB name: MTEB STSB config: default split: test revision: 0cde68302b3541bb8b3c340dc0644b0b745b3dc0 metrics: - type: cos_sim_pearson value: 51.354132781205294 - type: cos_sim_spearman value: 50.17494133468151 - type: euclidean_pearson value: 38.11014000059295 - type: euclidean_spearman value: 37.5997050693584 - type: manhattan_pearson value: 37.99036693507327 - type: manhattan_spearman value: 37.584133861146974 - task: type: STS dataset: type: mteb/stsbenchmark-sts name: MTEB STSBenchmark config: default split: test revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 metrics: - type: cos_sim_pearson value: 67.84222997484147 - type: cos_sim_spearman value: 67.39086924655895 - type: euclidean_pearson value: 51.87216665821589 - type: euclidean_spearman value: 51.309915218441695 - type: manhattan_pearson value: 51.871943884893454 - type: manhattan_spearman value: 51.271356979504375 - task: type: Classification dataset: type: ScandEval/scala-da name: MTEB ScalaDaClassification config: default split: test revision: 1de08520a7b361e92ffa2a2201ebd41942c54675 metrics: - type: accuracy value: 50.68359375 - type: ap value: 50.348173771768124 - type: f1 value: 50.38855675524828 - task: type: Classification dataset: type: ScandEval/scala-nb name: MTEB ScalaNbClassification config: default split: test revision: 237111a078ad5a834a55c57803d40bbe410ed03b metrics: - type: accuracy value: 50.9033203125 - type: ap value: 50.464534818390305 - type: f1 value: 50.12338348445107 - task: type: Classification dataset: type: ScandEval/scala-nn name: MTEB ScalaNnClassification config: default split: test revision: 9d9a2a4092ed3cacf0744592f6d2f32ab8ef4c0b metrics: - type: accuracy value: 50.7568359375 - type: ap value: 50.38550393534486 - type: f1 value: 50.19852110629762 - task: type: Classification dataset: type: ScandEval/scala-sv name: MTEB ScalaSvClassification config: default split: test revision: 1b48e3dcb02872335ff985ff938a054a4ed99008 metrics: - type: accuracy value: 50.9619140625 - type: ap value: 50.494690837095156 - type: f1 value: 50.26727387729569 - task: type: Reranking dataset: type: mteb/scidocs-reranking name: MTEB SciDocsRR config: default split: test revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab metrics: - type: map value: 68.36051662289248 - type: mrr value: 89.39224265204656 - task: type: Retrieval dataset: type: scifact name: MTEB SciFact config: default split: test revision: None metrics: - type: map_at_1 value: 23.721999999999998 - type: map_at_10 value: 31.335 - type: map_at_100 value: 32.461 - type: map_at_1000 value: 32.557 - type: map_at_3 value: 29.282000000000004 - type: map_at_5 value: 30.602 - type: mrr_at_1 value: 24.667 - type: mrr_at_10 value: 32.363 - type: mrr_at_100 value: 33.421 - type: mrr_at_1000 value: 33.499 - type: mrr_at_3 value: 30.444 - type: mrr_at_5 value: 31.628 - type: ndcg_at_1 value: 24.667 - type: ndcg_at_10 value: 35.29 - type: ndcg_at_100 value: 40.665 - type: ndcg_at_1000 value: 43.241 - type: ndcg_at_3 value: 31.238 - type: ndcg_at_5 value: 33.486 - type: precision_at_1 value: 24.667 - type: precision_at_10 value: 5.1 - type: precision_at_100 value: 0.7969999999999999 - type: precision_at_1000 value: 0.10300000000000001 - type: precision_at_3 value: 12.667 - type: precision_at_5 value: 8.933 - type: recall_at_1 value: 23.721999999999998 - type: recall_at_10 value: 46.417 - type: recall_at_100 value: 70.944 - type: recall_at_1000 value: 91.033 - type: recall_at_3 value: 35.693999999999996 - type: recall_at_5 value: 40.944 - task: type: PairClassification dataset: type: mteb/sprintduplicatequestions-pairclassification name: MTEB SprintDuplicateQuestions config: default split: test revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 metrics: - type: cos_sim_accuracy value: 99.5 - type: cos_sim_ap value: 77.07584309978081 - type: cos_sim_f1 value: 71.8864950078823 - type: cos_sim_precision value: 75.74750830564784 - type: cos_sim_recall value: 68.4 - type: dot_accuracy value: 99.19108910891089 - type: dot_ap value: 49.25491405967686 - type: dot_f1 value: 49.716932578486876 - type: dot_precision value: 51.21951219512195 - type: dot_recall value: 48.3 - type: euclidean_accuracy value: 99.19009900990099 - type: euclidean_ap value: 42.03570656308142 - type: euclidean_f1 value: 44.82153306026916 - type: euclidean_precision value: 54.01974612129761 - type: euclidean_recall value: 38.3 - type: manhattan_accuracy value: 99.19207920792078 - type: manhattan_ap value: 42.109436522707014 - type: manhattan_f1 value: 44.79905437352246 - type: manhattan_precision value: 54.76878612716764 - type: manhattan_recall value: 37.9 - type: max_accuracy value: 99.5 - type: max_ap value: 77.07584309978081 - type: max_f1 value: 71.8864950078823 - task: type: Clustering dataset: type: mteb/stackexchange-clustering name: MTEB StackExchangeClustering config: default split: test revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 metrics: - type: v_measure value: 37.40985053432638 - task: type: Clustering dataset: type: mteb/stackexchange-clustering-p2p name: MTEB StackExchangeClusteringP2P config: default split: test revision: 815ca46b2622cec33ccafc3735d572c266efdb44 metrics: - type: v_measure value: 19.657671371166966 - task: type: Reranking dataset: type: mteb/stackoverflowdupquestions-reranking name: MTEB StackOverflowDupQuestions config: default split: test revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 metrics: - type: map value: 38.612724125942385 - type: mrr value: 38.891130315762666 - task: type: Summarization dataset: type: mteb/summeval name: MTEB SummEval config: default split: test revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c metrics: - type: cos_sim_pearson value: 29.305333790118766 - type: cos_sim_spearman value: 30.556621737388685 - type: dot_pearson value: 30.309505984206115 - type: dot_spearman value: 31.302092660260872 - task: type: Classification dataset: type: ScandEval/swerec-mini name: MTEB SweRecClassification config: default split: test revision: 3c62f26bafdc4c4e1c16401ad4b32f0a94b46612 metrics: - type: accuracy value: 66.8701171875 - type: f1 value: 54.75017096649799 - task: type: Reranking dataset: type: C-MTEB/T2Reranking name: MTEB T2Reranking config: default split: dev revision: 76631901a18387f85eaa53e5450019b87ad58ef9 metrics: - type: map value: 55.34794621490011 - type: mrr value: 59.22764129348421 - task: type: Retrieval dataset: type: C-MTEB/T2Retrieval name: MTEB T2Retrieval config: default split: dev revision: 8731a845f1bf500a4f111cf1070785c793d10e64 metrics: - type: map_at_1 value: 0.586 - type: map_at_10 value: 0.819 - type: map_at_100 value: 0.8920000000000001 - type: map_at_1000 value: 0.928 - type: map_at_3 value: 0.729 - type: map_at_5 value: 0.771 - type: mrr_at_1 value: 1.9949999999999999 - type: mrr_at_10 value: 2.608 - type: mrr_at_100 value: 2.771 - type: mrr_at_1000 value: 2.8289999999999997 - type: mrr_at_3 value: 2.365 - type: mrr_at_5 value: 2.483 - type: ndcg_at_1 value: 1.9949999999999999 - type: ndcg_at_10 value: 1.314 - type: ndcg_at_100 value: 1.831 - type: ndcg_at_1000 value: 3.4139999999999997 - type: ndcg_at_3 value: 1.377 - type: ndcg_at_5 value: 1.2630000000000001 - type: precision_at_1 value: 1.9949999999999999 - type: precision_at_10 value: 0.488 - type: precision_at_100 value: 0.123 - type: precision_at_1000 value: 0.054 - type: precision_at_3 value: 1.027 - type: precision_at_5 value: 0.737 - type: recall_at_1 value: 0.586 - type: recall_at_10 value: 1.3390000000000002 - type: recall_at_100 value: 3.15 - type: recall_at_1000 value: 11.859 - type: recall_at_3 value: 0.8710000000000001 - type: recall_at_5 value: 1.0290000000000001 - task: type: Classification dataset: type: C-MTEB/TNews-classification name: MTEB TNews config: default split: validation revision: 317f262bf1e6126357bbe89e875451e4b0938fe4 metrics: - type: accuracy value: 37.254 - type: f1 value: 36.23675255530671 - task: type: Retrieval dataset: type: trec-covid name: MTEB TRECCOVID config: default split: test revision: None metrics: - type: map_at_1 value: 0.08499999999999999 - type: map_at_10 value: 0.462 - type: map_at_100 value: 0.893 - type: map_at_1000 value: 1.129 - type: map_at_3 value: 0.232 - type: map_at_5 value: 0.3 - type: mrr_at_1 value: 38.0 - type: mrr_at_10 value: 50.629999999999995 - type: mrr_at_100 value: 51.315999999999995 - type: mrr_at_1000 value: 51.365 - type: mrr_at_3 value: 47.0 - type: mrr_at_5 value: 48.9 - type: ndcg_at_1 value: 31.0 - type: ndcg_at_10 value: 24.823 - type: ndcg_at_100 value: 10.583 - type: ndcg_at_1000 value: 6.497999999999999 - type: ndcg_at_3 value: 30.95 - type: ndcg_at_5 value: 27.899 - type: precision_at_1 value: 38.0 - type: precision_at_10 value: 25.6 - type: precision_at_100 value: 8.98 - type: precision_at_1000 value: 2.248 - type: precision_at_3 value: 34.666999999999994 - type: precision_at_5 value: 29.599999999999998 - type: recall_at_1 value: 0.08499999999999999 - type: recall_at_10 value: 0.641 - type: recall_at_100 value: 2.002 - type: recall_at_1000 value: 4.902 - type: recall_at_3 value: 0.28200000000000003 - type: recall_at_5 value: 0.379 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (sqi-eng) config: sqi-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.9 - type: f1 value: 98.55000000000001 - type: precision value: 98.38333333333334 - type: recall value: 98.9 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fry-eng) config: fry-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 65.3179190751445 - type: f1 value: 59.44582071749702 - type: precision value: 57.49678869621066 - type: recall value: 65.3179190751445 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kur-eng) config: kur-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 38.53658536585366 - type: f1 value: 34.217555952803785 - type: precision value: 32.96511296649355 - type: recall value: 38.53658536585366 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tur-eng) config: tur-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.7 - type: f1 value: 98.26666666666665 - type: precision value: 98.05 - type: recall value: 98.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (deu-eng) config: deu-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 99.3 - type: f1 value: 99.13333333333333 - type: precision value: 99.05000000000001 - type: recall value: 99.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nld-eng) config: nld-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.89999999999999 - type: f1 value: 97.2 - type: precision value: 96.85000000000001 - type: recall value: 97.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ron-eng) config: ron-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.2 - type: f1 value: 97.6 - type: precision value: 97.3 - type: recall value: 98.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ang-eng) config: ang-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 52.23880597014925 - type: f1 value: 46.340992406389105 - type: precision value: 44.556384742951906 - type: recall value: 52.23880597014925 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ido-eng) config: ido-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.0 - type: f1 value: 93.67000000000002 - type: precision value: 93.075 - type: recall value: 95.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (jav-eng) config: jav-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 88.29268292682927 - type: f1 value: 85.76422764227642 - type: precision value: 84.84204413472706 - type: recall value: 88.29268292682927 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (isl-eng) config: isl-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.2 - type: f1 value: 96.46666666666667 - type: precision value: 96.1 - type: recall value: 97.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (slv-eng) config: slv-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.8408262454435 - type: f1 value: 95.9902794653706 - type: precision value: 95.56500607533415 - type: recall value: 96.8408262454435 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cym-eng) config: cym-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 93.3913043478261 - type: f1 value: 91.30434782608695 - type: precision value: 90.28985507246377 - type: recall value: 93.3913043478261 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kaz-eng) config: kaz-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 90.6086956521739 - type: f1 value: 88.1159420289855 - type: precision value: 86.9623188405797 - type: recall value: 90.6086956521739 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (est-eng) config: est-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.8 - type: f1 value: 97.16666666666667 - type: precision value: 96.86666666666667 - type: recall value: 97.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (heb-eng) config: heb-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 94.0 - type: f1 value: 92.34 - type: precision value: 91.54166666666667 - type: recall value: 94.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (gla-eng) config: gla-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 84.92159227985525 - type: f1 value: 80.8868975817106 - type: precision value: 79.11540008041817 - type: recall value: 84.92159227985525 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mar-eng) config: mar-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 94.89999999999999 - type: f1 value: 93.35 - type: precision value: 92.58333333333334 - type: recall value: 94.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lat-eng) config: lat-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 43.3 - type: f1 value: 36.64473116255726 - type: precision value: 34.64017752457381 - type: recall value: 43.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bel-eng) config: bel-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.7 - type: f1 value: 95.68333333333332 - type: precision value: 95.19999999999999 - type: recall value: 96.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pms-eng) config: pms-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 70.47619047619048 - type: f1 value: 66.63032734461308 - type: precision value: 65.46459191863879 - type: recall value: 70.47619047619048 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (gle-eng) config: gle-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 93.5 - type: f1 value: 91.63 - type: precision value: 90.75 - type: recall value: 93.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pes-eng) config: pes-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.5 - type: f1 value: 94.36666666666666 - type: precision value: 93.83333333333333 - type: recall value: 95.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nob-eng) config: nob-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 99.3 - type: f1 value: 99.06666666666666 - type: precision value: 98.95 - type: recall value: 99.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bul-eng) config: bul-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.8 - type: f1 value: 94.51666666666667 - type: precision value: 93.88333333333334 - type: recall value: 95.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cbk-eng) config: cbk-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 84.0 - type: f1 value: 80.46675324675326 - type: precision value: 78.95999999999998 - type: recall value: 84.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hun-eng) config: hun-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.7 - type: f1 value: 96.93333333333332 - type: precision value: 96.55 - type: recall value: 97.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (uig-eng) config: uig-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 92.10000000000001 - type: f1 value: 90.07333333333334 - type: precision value: 89.16166666666668 - type: recall value: 92.10000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (rus-eng) config: rus-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.6 - type: f1 value: 94.35 - type: precision value: 93.75 - type: recall value: 95.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (spa-eng) config: spa-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.9 - type: f1 value: 98.53333333333335 - type: precision value: 98.35000000000001 - type: recall value: 98.9 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hye-eng) config: hye-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.22641509433963 - type: f1 value: 95.14824797843666 - type: precision value: 94.60916442048517 - type: recall value: 96.22641509433963 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tel-eng) config: tel-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 93.58974358974359 - type: f1 value: 91.59544159544159 - type: precision value: 90.66951566951566 - type: recall value: 93.58974358974359 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (afr-eng) config: afr-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.1 - type: f1 value: 97.46666666666668 - type: precision value: 97.15 - type: recall value: 98.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mon-eng) config: mon-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 93.4090909090909 - type: f1 value: 91.5909090909091 - type: precision value: 90.71969696969697 - type: recall value: 93.4090909090909 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (arz-eng) config: arz-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 89.51781970649894 - type: f1 value: 86.76150544075072 - type: precision value: 85.55206149545772 - type: recall value: 89.51781970649894 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hrv-eng) config: hrv-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.2 - type: f1 value: 97.65 - type: precision value: 97.38333333333333 - type: recall value: 98.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nov-eng) config: nov-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 75.87548638132296 - type: f1 value: 71.24698906800073 - type: precision value: 69.66572338167668 - type: recall value: 75.87548638132296 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (gsw-eng) config: gsw-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 61.53846153846154 - type: f1 value: 54.83234714003944 - type: precision value: 52.06552706552707 - type: recall value: 61.53846153846154 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nds-eng) config: nds-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 59.199999999999996 - type: f1 value: 54.183211233211225 - type: precision value: 52.48751719986241 - type: recall value: 59.199999999999996 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ukr-eng) config: ukr-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.6 - type: f1 value: 94.3 - type: precision value: 93.65 - type: recall value: 95.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (uzb-eng) config: uzb-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 87.85046728971963 - type: f1 value: 85.25700934579439 - type: precision value: 84.09267912772586 - type: recall value: 87.85046728971963 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lit-eng) config: lit-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.0 - type: f1 value: 97.43333333333332 - type: precision value: 97.15 - type: recall value: 98.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ina-eng) config: ina-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 90.8 - type: f1 value: 88.66055555555555 - type: precision value: 87.81845238095238 - type: recall value: 90.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lfn-eng) config: lfn-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 70.6 - type: f1 value: 65.538895353013 - type: precision value: 63.69531394330308 - type: recall value: 70.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (zsm-eng) config: zsm-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.89999999999999 - type: f1 value: 96.06666666666668 - type: precision value: 95.68333333333334 - type: recall value: 96.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ita-eng) config: ita-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.8 - type: f1 value: 95.95 - type: precision value: 95.55 - type: recall value: 96.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cmn-eng) config: cmn-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.19999999999999 - type: f1 value: 93.8 - type: precision value: 93.13333333333334 - type: recall value: 95.19999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (lvs-eng) config: lvs-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.5 - type: f1 value: 95.45 - type: precision value: 94.93333333333334 - type: recall value: 96.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (glg-eng) config: glg-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.89999999999999 - type: f1 value: 97.28333333333332 - type: precision value: 96.98333333333333 - type: recall value: 97.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ceb-eng) config: ceb-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 78.16666666666666 - type: f1 value: 74.67336721249764 - type: precision value: 73.26035353535354 - type: recall value: 78.16666666666666 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bre-eng) config: bre-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 11.200000000000001 - type: f1 value: 8.48123815073815 - type: precision value: 7.843657708032708 - type: recall value: 11.200000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ben-eng) config: ben-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 91.3 - type: f1 value: 89.02333333333333 - type: precision value: 87.97500000000001 - type: recall value: 91.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (swg-eng) config: swg-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 72.32142857142857 - type: f1 value: 67.69209956709956 - type: precision value: 66.19047619047619 - type: recall value: 72.32142857142857 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (arq-eng) config: arq-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 79.69264544456641 - type: f1 value: 75.40693115885212 - type: precision value: 73.67544822539335 - type: recall value: 79.69264544456641 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kab-eng) config: kab-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 86.8 - type: f1 value: 83.65666666666667 - type: precision value: 82.24833333333333 - type: recall value: 86.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fra-eng) config: fra-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.39999999999999 - type: f1 value: 95.36666666666666 - type: precision value: 94.86666666666666 - type: recall value: 96.39999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (por-eng) config: por-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.3 - type: f1 value: 95.49 - type: precision value: 95.10833333333333 - type: recall value: 96.3 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tat-eng) config: tat-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 89.60000000000001 - type: f1 value: 87.04746031746032 - type: precision value: 85.89583333333333 - type: recall value: 89.60000000000001 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (oci-eng) config: oci-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 86.9 - type: f1 value: 84.57088023088022 - type: precision value: 83.6475 - type: recall value: 86.9 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pol-eng) config: pol-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.2 - type: f1 value: 97.7 - type: precision value: 97.46666666666668 - type: recall value: 98.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (war-eng) config: war-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 85.39999999999999 - type: f1 value: 82.83333333333333 - type: precision value: 81.80137426900586 - type: recall value: 85.39999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (aze-eng) config: aze-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 91.4 - type: f1 value: 89.11999999999999 - type: precision value: 88.12777777777778 - type: recall value: 91.4 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (vie-eng) config: vie-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.8 - type: f1 value: 97.16666666666669 - type: precision value: 96.85000000000001 - type: recall value: 97.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (nno-eng) config: nno-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.89999999999999 - type: f1 value: 97.30666666666666 - type: precision value: 97.02499999999999 - type: recall value: 97.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cha-eng) config: cha-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 27.00729927007299 - type: f1 value: 25.114895917815623 - type: precision value: 24.602283361407448 - type: recall value: 27.00729927007299 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mhr-eng) config: mhr-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 14.099999999999998 - type: f1 value: 11.869284007509814 - type: precision value: 11.199695454818405 - type: recall value: 14.099999999999998 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (dan-eng) config: dan-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.7 - type: f1 value: 97.09 - type: precision value: 96.80833333333332 - type: recall value: 97.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ell-eng) config: ell-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.5 - type: f1 value: 95.47333333333333 - type: precision value: 94.975 - type: recall value: 96.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (amh-eng) config: amh-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 93.45238095238095 - type: f1 value: 91.66666666666666 - type: precision value: 90.77380952380952 - type: recall value: 93.45238095238095 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (pam-eng) config: pam-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 11.899999999999999 - type: f1 value: 10.303261315113037 - type: precision value: 9.902986584515606 - type: recall value: 11.899999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hsb-eng) config: hsb-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 81.57349896480332 - type: f1 value: 77.86519438693352 - type: precision value: 76.35595081247254 - type: recall value: 81.57349896480332 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (srp-eng) config: srp-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.1 - type: f1 value: 94.86666666666667 - type: precision value: 94.25 - type: recall value: 96.1 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (epo-eng) config: epo-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.8 - type: f1 value: 98.46666666666667 - type: precision value: 98.3 - type: recall value: 98.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kzj-eng) config: kzj-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 10.7 - type: f1 value: 8.621683883854935 - type: precision value: 8.188292731521031 - type: recall value: 10.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (awa-eng) config: awa-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 90.47619047619048 - type: f1 value: 87.8581735724593 - type: precision value: 86.72438672438673 - type: recall value: 90.47619047619048 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fao-eng) config: fao-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.0381679389313 - type: f1 value: 93.60050890585242 - type: precision value: 92.970737913486 - type: recall value: 95.0381679389313 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mal-eng) config: mal-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.2532751091703 - type: f1 value: 97.67103347889375 - type: precision value: 97.37991266375546 - type: recall value: 98.2532751091703 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ile-eng) config: ile-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 84.6 - type: f1 value: 80.99904761904763 - type: precision value: 79.54634920634919 - type: recall value: 84.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (bos-eng) config: bos-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.89265536723164 - type: f1 value: 95.90395480225989 - type: precision value: 95.4331450094162 - type: recall value: 96.89265536723164 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cor-eng) config: cor-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 12.6 - type: f1 value: 9.981918087824628 - type: precision value: 9.326319147606549 - type: recall value: 12.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (cat-eng) config: cat-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.39999999999999 - type: f1 value: 96.65 - type: precision value: 96.28333333333333 - type: recall value: 97.39999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (eus-eng) config: eus-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.5 - type: f1 value: 95.38333333333333 - type: precision value: 94.83333333333333 - type: recall value: 96.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (yue-eng) config: yue-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 90.8 - type: f1 value: 88.43666666666665 - type: precision value: 87.395 - type: recall value: 90.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (swe-eng) config: swe-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.7 - type: f1 value: 97.03333333333333 - type: precision value: 96.71666666666667 - type: recall value: 97.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (dtp-eng) config: dtp-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 9.4 - type: f1 value: 7.946889105220061 - type: precision value: 7.665059865752875 - type: recall value: 9.4 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kat-eng) config: kat-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.04021447721179 - type: f1 value: 93.68632707774799 - type: precision value: 93.08534405719392 - type: recall value: 95.04021447721179 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (jpn-eng) config: jpn-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.89999999999999 - type: f1 value: 94.66666666666667 - type: precision value: 94.08333333333334 - type: recall value: 95.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (csb-eng) config: csb-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 82.6086956521739 - type: f1 value: 77.98418972332016 - type: precision value: 75.96837944664031 - type: recall value: 82.6086956521739 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (xho-eng) config: xho-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.77464788732394 - type: f1 value: 94.8356807511737 - type: precision value: 94.36619718309859 - type: recall value: 95.77464788732394 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (orv-eng) config: orv-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 53.17365269461077 - type: f1 value: 47.07043056743655 - type: precision value: 45.161363241830784 - type: recall value: 53.17365269461077 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ind-eng) config: ind-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.5 - type: f1 value: 94.5 - type: precision value: 94.03333333333333 - type: recall value: 95.5 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tuk-eng) config: tuk-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 93.59605911330048 - type: f1 value: 91.82266009852216 - type: precision value: 91.09195402298852 - type: recall value: 93.59605911330048 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (max-eng) config: max-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 76.40845070422534 - type: f1 value: 72.73082942097027 - type: precision value: 71.46686939820742 - type: recall value: 76.40845070422534 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (swh-eng) config: swh-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 93.58974358974359 - type: f1 value: 91.98290598290598 - type: precision value: 91.3119658119658 - type: recall value: 93.58974358974359 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (hin-eng) config: hin-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.8 - type: f1 value: 97.06666666666668 - type: precision value: 96.7 - type: recall value: 97.8 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (dsb-eng) config: dsb-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 68.89352818371609 - type: f1 value: 64.47860652453555 - type: precision value: 62.878651918592574 - type: recall value: 68.89352818371609 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ber-eng) config: ber-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 33.800000000000004 - type: f1 value: 29.290774344112368 - type: precision value: 28.066016735704647 - type: recall value: 33.800000000000004 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tam-eng) config: tam-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 90.22801302931596 - type: f1 value: 88.07817589576547 - type: precision value: 87.171552660152 - type: recall value: 90.22801302931596 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (slk-eng) config: slk-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.2 - type: f1 value: 97.63333333333334 - type: precision value: 97.36666666666667 - type: recall value: 98.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tgl-eng) config: tgl-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.7 - type: f1 value: 96.95 - type: precision value: 96.58333333333331 - type: recall value: 97.7 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ast-eng) config: ast-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 92.91338582677166 - type: f1 value: 90.81364829396327 - type: precision value: 89.89501312335958 - type: recall value: 92.91338582677166 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (mkd-eng) config: mkd-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.89999999999999 - type: f1 value: 95.98333333333332 - type: precision value: 95.56666666666668 - type: recall value: 96.89999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (khm-eng) config: khm-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 74.51523545706371 - type: f1 value: 70.20346919931407 - type: precision value: 68.6389565788895 - type: recall value: 74.51523545706371 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ces-eng) config: ces-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 97.6 - type: f1 value: 96.88333333333333 - type: precision value: 96.53333333333333 - type: recall value: 97.6 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tzl-eng) config: tzl-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 46.15384615384615 - type: f1 value: 39.47885447885448 - type: precision value: 37.301528599605525 - type: recall value: 46.15384615384615 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (urd-eng) config: urd-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 94.69999999999999 - type: f1 value: 93.16666666666667 - type: precision value: 92.41666666666667 - type: recall value: 94.69999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (ara-eng) config: ara-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.19999999999999 - type: f1 value: 93.83333333333333 - type: precision value: 93.16666666666667 - type: recall value: 95.19999999999999 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (kor-eng) config: kor-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 92.0 - type: f1 value: 89.98666666666666 - type: precision value: 89.09166666666667 - type: recall value: 92.0 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (yid-eng) config: yid-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 95.51886792452831 - type: f1 value: 94.3003144654088 - type: precision value: 93.75 - type: recall value: 95.51886792452831 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (fin-eng) config: fin-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 98.2 - type: f1 value: 97.83333333333333 - type: precision value: 97.65 - type: recall value: 98.2 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (tha-eng) config: tha-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 96.8978102189781 - type: f1 value: 96.04622871046227 - type: precision value: 95.62043795620438 - type: recall value: 96.8978102189781 - task: type: BitextMining dataset: type: mteb/tatoeba-bitext-mining name: MTEB Tatoeba (wuu-eng) config: wuu-eng split: test revision: 9080400076fbadbb4c4dcb136ff4eddc40b42553 metrics: - type: accuracy value: 85.1 - type: f1 value: 81.78564213564214 - type: precision value: 80.46416666666667 - type: recall value: 85.1 - task: type: Clustering dataset: type: slvnwhrl/tenkgnad-clustering-p2p name: MTEB TenKGnadClusteringP2P config: default split: test revision: 5c59e41555244b7e45c9a6be2d720ab4bafae558 metrics: - type: v_measure value: 19.988360713728564 - task: type: Clustering dataset: type: slvnwhrl/tenkgnad-clustering-s2s name: MTEB TenKGnadClusteringS2S config: default split: test revision: 6cddbe003f12b9b140aec477b583ac4191f01786 metrics: - type: v_measure value: 25.49952010397176 - task: type: Clustering dataset: type: C-MTEB/ThuNewsClusteringP2P name: MTEB ThuNewsClusteringP2P config: default split: test revision: 5798586b105c0434e4f0fe5e767abe619442cf93 metrics: - type: v_measure value: 32.012653950111556 - task: type: Clustering dataset: type: C-MTEB/ThuNewsClusteringS2S name: MTEB ThuNewsClusteringS2S config: default split: test revision: 8a8b2caeda43f39e13c4bc5bea0f8a667896e10d metrics: - type: v_measure value: 42.08914091890925 - task: type: Retrieval dataset: type: webis-touche2020 name: MTEB Touche2020 config: default split: test revision: None metrics: - type: map_at_1 value: 0.20500000000000002 - type: map_at_10 value: 0.391 - type: map_at_100 value: 0.612 - type: map_at_1000 value: 0.645 - type: map_at_3 value: 0.302 - type: map_at_5 value: 0.383 - type: mrr_at_1 value: 4.082 - type: mrr_at_10 value: 5.612 - type: mrr_at_100 value: 6.822 - type: mrr_at_1000 value: 6.929 - type: mrr_at_3 value: 4.082 - type: mrr_at_5 value: 5.408 - type: ndcg_at_1 value: 4.082 - type: ndcg_at_10 value: 1.6840000000000002 - type: ndcg_at_100 value: 2.876 - type: ndcg_at_1000 value: 4.114 - type: ndcg_at_3 value: 2.52 - type: ndcg_at_5 value: 2.3720000000000003 - type: precision_at_1 value: 4.082 - type: precision_at_10 value: 1.429 - type: precision_at_100 value: 0.755 - type: precision_at_1000 value: 0.18 - type: precision_at_3 value: 2.041 - type: precision_at_5 value: 2.4490000000000003 - type: recall_at_1 value: 0.20500000000000002 - type: recall_at_10 value: 0.761 - type: recall_at_100 value: 4.423 - type: recall_at_1000 value: 9.044 - type: recall_at_3 value: 0.302 - type: recall_at_5 value: 0.683 - task: type: Classification dataset: type: mteb/toxic_conversations_50k name: MTEB ToxicConversationsClassification config: default split: test revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c metrics: - type: accuracy value: 63.5394 - type: ap value: 11.440330234311777 - type: f1 value: 48.937405112008854 - task: type: Classification dataset: type: mteb/tweet_sentiment_extraction name: MTEB TweetSentimentExtractionClassification config: default split: test revision: d604517c81ca91fe16a244d1248fc021f9ecee7a metrics: - type: accuracy value: 55.16694963214488 - type: f1 value: 55.275387066931124 - task: type: Clustering dataset: type: mteb/twentynewsgroups-clustering name: MTEB TwentyNewsgroupsClustering config: default split: test revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 metrics: - type: v_measure value: 19.628551067284956 - task: type: PairClassification dataset: type: mteb/twittersemeval2015-pairclassification name: MTEB TwitterSemEval2015 config: default split: test revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 metrics: - type: cos_sim_accuracy value: 80.3302139834297 - type: cos_sim_ap value: 53.577244820707605 - type: cos_sim_f1 value: 51.58639580004565 - type: cos_sim_precision value: 45.45454545454545 - type: cos_sim_recall value: 59.63060686015831 - type: dot_accuracy value: 80.14543720569827 - type: dot_ap value: 52.0875752603294 - type: dot_f1 value: 51.38086062941555 - type: dot_precision value: 43.22766570605187 - type: dot_recall value: 63.3245382585752 - type: euclidean_accuracy value: 78.82815759670979 - type: euclidean_ap value: 42.100001806860824 - type: euclidean_f1 value: 42.769230769230774 - type: euclidean_precision value: 34.98322147651007 - type: euclidean_recall value: 55.01319261213721 - type: manhattan_accuracy value: 78.82815759670979 - type: manhattan_ap value: 41.936649509973165 - type: manhattan_f1 value: 42.57720324202319 - type: manhattan_precision value: 34.832969615578314 - type: manhattan_recall value: 54.74934036939314 - type: max_accuracy value: 80.3302139834297 - type: max_ap value: 53.577244820707605 - type: max_f1 value: 51.58639580004565 - task: type: PairClassification dataset: type: mteb/twitterurlcorpus-pairclassification name: MTEB TwitterURLCorpus config: default split: test revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf metrics: - type: cos_sim_accuracy value: 87.45876508712695 - type: cos_sim_ap value: 83.53208334948403 - type: cos_sim_f1 value: 75.54560716284276 - type: cos_sim_precision value: 73.27929362379678 - type: cos_sim_recall value: 77.95657530027718 - type: dot_accuracy value: 85.64054798773626 - type: dot_ap value: 77.96456929342345 - type: dot_f1 value: 72.09775967413442 - type: dot_precision value: 67.26448429895326 - type: dot_recall value: 77.6793963658762 - type: euclidean_accuracy value: 84.78480226646485 - type: euclidean_ap value: 75.64618087898923 - type: euclidean_f1 value: 67.1075581395349 - type: euclidean_precision value: 63.54252683732452 - type: euclidean_recall value: 71.0963966738528 - type: manhattan_accuracy value: 84.72658827182055 - type: manhattan_ap value: 75.51009726264691 - type: manhattan_f1 value: 67.00128311570685 - type: manhattan_precision value: 65.70688378978534 - type: manhattan_recall value: 68.34770557437635 - type: max_accuracy value: 87.45876508712695 - type: max_ap value: 83.53208334948403 - type: max_f1 value: 75.54560716284276 - task: type: Retrieval dataset: type: C-MTEB/VideoRetrieval name: MTEB VideoRetrieval config: default split: dev revision: 58c2597a5943a2ba48f4668c3b90d796283c5639 metrics: - type: map_at_1 value: 13.100000000000001 - type: map_at_10 value: 15.620000000000001 - type: map_at_100 value: 15.928 - type: map_at_1000 value: 15.976 - type: map_at_3 value: 14.817 - type: map_at_5 value: 15.322 - type: mrr_at_1 value: 13.0 - type: mrr_at_10 value: 15.57 - type: mrr_at_100 value: 15.878 - type: mrr_at_1000 value: 15.926000000000002 - type: mrr_at_3 value: 14.767 - type: mrr_at_5 value: 15.272 - type: ndcg_at_1 value: 13.100000000000001 - type: ndcg_at_10 value: 17.05 - type: ndcg_at_100 value: 18.801000000000002 - type: ndcg_at_1000 value: 20.436 - type: ndcg_at_3 value: 15.425 - type: ndcg_at_5 value: 16.333000000000002 - type: precision_at_1 value: 13.100000000000001 - type: precision_at_10 value: 2.16 - type: precision_at_100 value: 0.304 - type: precision_at_1000 value: 0.044000000000000004 - type: precision_at_3 value: 5.733 - type: precision_at_5 value: 3.88 - type: recall_at_1 value: 13.100000000000001 - type: recall_at_10 value: 21.6 - type: recall_at_100 value: 30.4 - type: recall_at_1000 value: 44.1 - type: recall_at_3 value: 17.2 - type: recall_at_5 value: 19.400000000000002 - task: type: Classification dataset: type: C-MTEB/waimai-classification name: MTEB Waimai config: default split: test revision: 339287def212450dcaa9df8c22bf93e9980c7023 metrics: - type: accuracy value: 70.44 - type: ap value: 49.70788956752899 - type: f1 value: 69.57867775339233 - task: type: Clustering dataset: type: jinaai/cities_wiki_clustering name: MTEB WikiCitiesClustering config: default split: test revision: ddc9ee9242fa65332597f70e967ecc38b9d734fa metrics: - type: v_measure value: 48.038657353672534 --- # SONAR [[Paper]](https://ai.meta.com/research/publications/sonar-sentence-level-multimodal-and-language-agnostic-representations/) We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space, with a full suite of speech and text encoders and decoders. It substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. We also provide a single text decoder, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. *SONAR* stands for **S**entence-level multim**O**dal and la**N**guage-**A**gnostic **R**epresentations The full list of supported languages (along with download links) can be found here [below](#supported-languages-and-download-links). ## Installing SONAR depends mainly on [Fairseq2](https://github.com/fairinternal/fairseq2) and can be installed using (tested with `python=3.8`) ```bash pip install --upgrade pip pip config set global.extra-index-url https://test.pypi.org/simple/ pip install -e . ``` ## Usage fairseq2 will automatically download models into your `$TORCH_HOME/hub` directory upon using the commands below. ### Compute text sentence embeddings with SONAR: ```python from sonar.inference_pipelines.text import TextToEmbeddingModelPipeline t2vec_model = TextToEmbeddingModelPipeline(encoder="text_sonar_basic_encoder", tokenizer="text_sonar_basic_encoder") sentences = ['My name is SONAR.', 'I can embed the sentences into vectorial space.'] t2vec_model.predict(sentences, source_lang="eng_Latn").shape # torch.Size([2, 1024]) ``` ### Translate text with SONAR ```python from sonar.inference_pipelines.text import TextToTextModelPipeline t2t_model = TextToTextModelPipeline(encoder="text_sonar_basic_encoder", decoder="text_sonar_basic_decoder", tokenizer="text_sonar_basic_encoder") # tokenizer is attached to both encoder and decoder cards sentences = ['My name is SONAR.', 'I can embed the sentences into vectorial space.'] t2t_model.predict(sentences, source_lang="eng_Latn", target_lang="fra_Latn") # ['Mon nom est SONAR.', "Je peux intégrer les phrases dans l'espace vectoriel."] ``` ### Compute speech sentence embeddings with SONAR ```python from sonar.inference_pipelines.speech import SpeechToEmbeddingModelPipeline s2vec_model = SpeechToEmbeddingModelPipeline(encoder="sonar_speech_encoder_eng") s2vec_model.predict(["./tests/integration_tests/data/audio_files/audio_1.wav", "./tests/integration_tests/data/audio_files/audio_2.wav"]).shape # torch.Size([2, 1024]) import torchaudio inp, sr = torchaudio.load("./tests/integration_tests/data/audio_files/audio_1.wav") assert sr == 16000, "Sample rate should be 16kHz" s2vec_model.predict([inp]).shape # torch.Size([1, 1024]) ``` ### Speech-to-text translation with SONAR ```python from sonar.inference_pipelines.speech import SpeechToTextModelPipeline s2t_model = SpeechToTextModelPipeline(encoder="sonar_speech_encoder_eng", decoder="text_sonar_basic_decoder", tokenizer="text_sonar_basic_decoder") import torchaudio inp, sr = torchaudio.load("./tests/integration_tests/data/audio_files/audio_1.wav") assert sr == 16000, "Sample rate should be 16kHz" # passing loaded audio files s2t_model.predict([inp], target_lang="eng_Latn") # ['Television reports show white smoke coming from the plant.'] # passing multiple wav files s2t_model.predict(["./tests/integration_tests/data/audio_files/audio_1.wav", "./tests/integration_tests/data/audio_files/audio_2.wav"], target_lang="eng_Latn") # ['Television reports show white smoke coming from the plant.', # 'These couples may choose to make an adoption plan for their baby.'] ``` ### Predicting [cross-lingual semantic similarity](https://github.com/facebookresearch/fairseq/tree/nllb/examples/nllb/human_XSTS_eval) with BLASER 2 models ```Python import torch from sonar.models.blaser.loader import load_blaser_model blaser_ref = load_blaser_model("blaser_st2st_ref_v2_0").eval() blaser_qe = load_blaser_model("blaser_st2st_qe_v2_0").eval() # BLASER-2 is supposed to work with SONAR speech and text embeddings, # but we didn't include their extraction in this snippet, to keep it simple. emb = torch.ones([1, 1024]) print(blaser_ref(src=emb, ref=emb, mt=emb).item()) # 5.2552 print(blaser_qe(src=emb, mt=emb).item()) # 4.9819 ``` See more complete demo notebooks : * [sonar text2text similarity and translation](examples/sonar_text_demo.ipynb) * [sonar speech2text and other data pipeline examples](examples/inference_pipelines.ipynb) ## Model details - **Developed by:** Paul-Ambroise Duquenne et al. - **License:** CC-BY-NC 4.0 license - **Cite as:** ``` @article{Duquenne:2023:sonar_arxiv, author = {Paul-Ambroise Duquenne and Holger Schwenk and Benoit Sagot}, title = {{SONAR:} Sentence-Level Multimodal and Language-Agnostic Representations}, publisher = {arXiv}, year = {2023}, url = {https://arxiv.org/abs/unk}, } ```