|
--- |
|
tags: |
|
- ctranslate2 |
|
- int8 |
|
- float16 |
|
- mteb |
|
model-index: |
|
- name: e5-small-v2 |
|
results: |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_counterfactual |
|
name: MTEB AmazonCounterfactualClassification (en) |
|
config: en |
|
split: test |
|
revision: e8379541af4e31359cca9fbcf4b00f2671dba205 |
|
metrics: |
|
- type: accuracy |
|
value: 77.59701492537313 |
|
- type: ap |
|
value: 41.67064885731708 |
|
- type: f1 |
|
value: 71.86465946398573 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_polarity |
|
name: MTEB AmazonPolarityClassification |
|
config: default |
|
split: test |
|
revision: e2d317d38cd51312af73b3d32a06d1a08b442046 |
|
metrics: |
|
- type: accuracy |
|
value: 91.265875 |
|
- type: ap |
|
value: 87.67633085349644 |
|
- type: f1 |
|
value: 91.24297521425744 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_reviews_multi |
|
name: MTEB AmazonReviewsClassification (en) |
|
config: en |
|
split: test |
|
revision: 1399c76144fd37290681b995c656ef9b2e06e26d |
|
metrics: |
|
- type: accuracy |
|
value: 45.882000000000005 |
|
- type: f1 |
|
value: 45.08058870381236 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: arguana |
|
name: MTEB ArguAna |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 20.697 |
|
- type: map_at_10 |
|
value: 33.975 |
|
- type: map_at_100 |
|
value: 35.223 |
|
- type: map_at_1000 |
|
value: 35.260000000000005 |
|
- type: map_at_3 |
|
value: 29.776999999999997 |
|
- type: map_at_5 |
|
value: 32.035000000000004 |
|
- type: mrr_at_1 |
|
value: 20.982 |
|
- type: mrr_at_10 |
|
value: 34.094 |
|
- type: mrr_at_100 |
|
value: 35.343 |
|
- type: mrr_at_1000 |
|
value: 35.38 |
|
- type: mrr_at_3 |
|
value: 29.884 |
|
- type: mrr_at_5 |
|
value: 32.141999999999996 |
|
- type: ndcg_at_1 |
|
value: 20.697 |
|
- type: ndcg_at_10 |
|
value: 41.668 |
|
- type: ndcg_at_100 |
|
value: 47.397 |
|
- type: ndcg_at_1000 |
|
value: 48.305 |
|
- type: ndcg_at_3 |
|
value: 32.928000000000004 |
|
- type: ndcg_at_5 |
|
value: 36.998999999999995 |
|
- type: precision_at_1 |
|
value: 20.697 |
|
- type: precision_at_10 |
|
value: 6.636 |
|
- type: precision_at_100 |
|
value: 0.924 |
|
- type: precision_at_1000 |
|
value: 0.099 |
|
- type: precision_at_3 |
|
value: 14.035 |
|
- type: precision_at_5 |
|
value: 10.398 |
|
- type: recall_at_1 |
|
value: 20.697 |
|
- type: recall_at_10 |
|
value: 66.35799999999999 |
|
- type: recall_at_100 |
|
value: 92.39 |
|
- type: recall_at_1000 |
|
value: 99.36 |
|
- type: recall_at_3 |
|
value: 42.105 |
|
- type: recall_at_5 |
|
value: 51.991 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-p2p |
|
name: MTEB ArxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d |
|
metrics: |
|
- type: v_measure |
|
value: 42.1169517447068 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/arxiv-clustering-s2s |
|
name: MTEB ArxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53 |
|
metrics: |
|
- type: v_measure |
|
value: 34.79553720107097 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/askubuntudupquestions-reranking |
|
name: MTEB AskUbuntuDupQuestions |
|
config: default |
|
split: test |
|
revision: 2000358ca161889fa9c082cb41daa8dcfb161a54 |
|
metrics: |
|
- type: map |
|
value: 58.10811337308168 |
|
- type: mrr |
|
value: 71.56410763751482 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/biosses-sts |
|
name: MTEB BIOSSES |
|
config: default |
|
split: test |
|
revision: d3fb88f8f02e40887cd149695127462bbcf29b4a |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 78.46834918248696 |
|
- type: cos_sim_spearman |
|
value: 79.4289182755206 |
|
- type: euclidean_pearson |
|
value: 76.26662973727008 |
|
- type: euclidean_spearman |
|
value: 78.11744260952536 |
|
- type: manhattan_pearson |
|
value: 76.08175262609434 |
|
- type: manhattan_spearman |
|
value: 78.29395265552289 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/banking77 |
|
name: MTEB Banking77Classification |
|
config: default |
|
split: test |
|
revision: 0fd18e25b25c072e09e0d92ab615fda904d66300 |
|
metrics: |
|
- type: accuracy |
|
value: 81.63636363636364 |
|
- type: f1 |
|
value: 81.55779952376953 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-p2p |
|
name: MTEB BiorxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40 |
|
metrics: |
|
- type: v_measure |
|
value: 35.88541137137571 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/biorxiv-clustering-s2s |
|
name: MTEB BiorxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908 |
|
metrics: |
|
- type: v_measure |
|
value: 30.05205685274407 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackAndroidRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 30.293999999999997 |
|
- type: map_at_10 |
|
value: 39.876 |
|
- type: map_at_100 |
|
value: 41.315000000000005 |
|
- type: map_at_1000 |
|
value: 41.451 |
|
- type: map_at_3 |
|
value: 37.194 |
|
- type: map_at_5 |
|
value: 38.728 |
|
- type: mrr_at_1 |
|
value: 37.053000000000004 |
|
- type: mrr_at_10 |
|
value: 45.281 |
|
- type: mrr_at_100 |
|
value: 46.188 |
|
- type: mrr_at_1000 |
|
value: 46.245999999999995 |
|
- type: mrr_at_3 |
|
value: 43.228 |
|
- type: mrr_at_5 |
|
value: 44.366 |
|
- type: ndcg_at_1 |
|
value: 37.053000000000004 |
|
- type: ndcg_at_10 |
|
value: 45.086 |
|
- type: ndcg_at_100 |
|
value: 50.756 |
|
- type: ndcg_at_1000 |
|
value: 53.123 |
|
- type: ndcg_at_3 |
|
value: 41.416 |
|
- type: ndcg_at_5 |
|
value: 43.098 |
|
- type: precision_at_1 |
|
value: 37.053000000000004 |
|
- type: precision_at_10 |
|
value: 8.34 |
|
- type: precision_at_100 |
|
value: 1.346 |
|
- type: precision_at_1000 |
|
value: 0.186 |
|
- type: precision_at_3 |
|
value: 19.647000000000002 |
|
- type: precision_at_5 |
|
value: 13.877 |
|
- type: recall_at_1 |
|
value: 30.293999999999997 |
|
- type: recall_at_10 |
|
value: 54.309 |
|
- type: recall_at_100 |
|
value: 78.59 |
|
- type: recall_at_1000 |
|
value: 93.82300000000001 |
|
- type: recall_at_3 |
|
value: 43.168 |
|
- type: recall_at_5 |
|
value: 48.192 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackEnglishRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 28.738000000000003 |
|
- type: map_at_10 |
|
value: 36.925999999999995 |
|
- type: map_at_100 |
|
value: 38.017 |
|
- type: map_at_1000 |
|
value: 38.144 |
|
- type: map_at_3 |
|
value: 34.446 |
|
- type: map_at_5 |
|
value: 35.704 |
|
- type: mrr_at_1 |
|
value: 35.478 |
|
- type: mrr_at_10 |
|
value: 42.786 |
|
- type: mrr_at_100 |
|
value: 43.458999999999996 |
|
- type: mrr_at_1000 |
|
value: 43.507 |
|
- type: mrr_at_3 |
|
value: 40.648 |
|
- type: mrr_at_5 |
|
value: 41.804 |
|
- type: ndcg_at_1 |
|
value: 35.478 |
|
- type: ndcg_at_10 |
|
value: 42.044 |
|
- type: ndcg_at_100 |
|
value: 46.249 |
|
- type: ndcg_at_1000 |
|
value: 48.44 |
|
- type: ndcg_at_3 |
|
value: 38.314 |
|
- type: ndcg_at_5 |
|
value: 39.798 |
|
- type: precision_at_1 |
|
value: 35.478 |
|
- type: precision_at_10 |
|
value: 7.764 |
|
- type: precision_at_100 |
|
value: 1.253 |
|
- type: precision_at_1000 |
|
value: 0.174 |
|
- type: precision_at_3 |
|
value: 18.047 |
|
- type: precision_at_5 |
|
value: 12.637 |
|
- type: recall_at_1 |
|
value: 28.738000000000003 |
|
- type: recall_at_10 |
|
value: 50.659 |
|
- type: recall_at_100 |
|
value: 68.76299999999999 |
|
- type: recall_at_1000 |
|
value: 82.811 |
|
- type: recall_at_3 |
|
value: 39.536 |
|
- type: recall_at_5 |
|
value: 43.763999999999996 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGamingRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 38.565 |
|
- type: map_at_10 |
|
value: 50.168 |
|
- type: map_at_100 |
|
value: 51.11 |
|
- type: map_at_1000 |
|
value: 51.173 |
|
- type: map_at_3 |
|
value: 47.044000000000004 |
|
- type: map_at_5 |
|
value: 48.838 |
|
- type: mrr_at_1 |
|
value: 44.201 |
|
- type: mrr_at_10 |
|
value: 53.596999999999994 |
|
- type: mrr_at_100 |
|
value: 54.211 |
|
- type: mrr_at_1000 |
|
value: 54.247 |
|
- type: mrr_at_3 |
|
value: 51.202000000000005 |
|
- type: mrr_at_5 |
|
value: 52.608999999999995 |
|
- type: ndcg_at_1 |
|
value: 44.201 |
|
- type: ndcg_at_10 |
|
value: 55.694 |
|
- type: ndcg_at_100 |
|
value: 59.518 |
|
- type: ndcg_at_1000 |
|
value: 60.907 |
|
- type: ndcg_at_3 |
|
value: 50.395999999999994 |
|
- type: ndcg_at_5 |
|
value: 53.022999999999996 |
|
- type: precision_at_1 |
|
value: 44.201 |
|
- type: precision_at_10 |
|
value: 8.84 |
|
- type: precision_at_100 |
|
value: 1.162 |
|
- type: precision_at_1000 |
|
value: 0.133 |
|
- type: precision_at_3 |
|
value: 22.153 |
|
- type: precision_at_5 |
|
value: 15.260000000000002 |
|
- type: recall_at_1 |
|
value: 38.565 |
|
- type: recall_at_10 |
|
value: 68.65 |
|
- type: recall_at_100 |
|
value: 85.37400000000001 |
|
- type: recall_at_1000 |
|
value: 95.37400000000001 |
|
- type: recall_at_3 |
|
value: 54.645999999999994 |
|
- type: recall_at_5 |
|
value: 60.958 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackGisRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.945 |
|
- type: map_at_10 |
|
value: 30.641000000000002 |
|
- type: map_at_100 |
|
value: 31.599 |
|
- type: map_at_1000 |
|
value: 31.691000000000003 |
|
- type: map_at_3 |
|
value: 28.405 |
|
- type: map_at_5 |
|
value: 29.704000000000004 |
|
- type: mrr_at_1 |
|
value: 25.537 |
|
- type: mrr_at_10 |
|
value: 32.22 |
|
- type: mrr_at_100 |
|
value: 33.138 |
|
- type: mrr_at_1000 |
|
value: 33.214 |
|
- type: mrr_at_3 |
|
value: 30.151 |
|
- type: mrr_at_5 |
|
value: 31.298 |
|
- type: ndcg_at_1 |
|
value: 25.537 |
|
- type: ndcg_at_10 |
|
value: 34.638000000000005 |
|
- type: ndcg_at_100 |
|
value: 39.486 |
|
- type: ndcg_at_1000 |
|
value: 41.936 |
|
- type: ndcg_at_3 |
|
value: 30.333 |
|
- type: ndcg_at_5 |
|
value: 32.482 |
|
- type: precision_at_1 |
|
value: 25.537 |
|
- type: precision_at_10 |
|
value: 5.153 |
|
- type: precision_at_100 |
|
value: 0.7929999999999999 |
|
- type: precision_at_1000 |
|
value: 0.104 |
|
- type: precision_at_3 |
|
value: 12.429 |
|
- type: precision_at_5 |
|
value: 8.723 |
|
- type: recall_at_1 |
|
value: 23.945 |
|
- type: recall_at_10 |
|
value: 45.412 |
|
- type: recall_at_100 |
|
value: 67.836 |
|
- type: recall_at_1000 |
|
value: 86.467 |
|
- type: recall_at_3 |
|
value: 34.031 |
|
- type: recall_at_5 |
|
value: 39.039 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackMathematicaRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 14.419 |
|
- type: map_at_10 |
|
value: 20.858999999999998 |
|
- type: map_at_100 |
|
value: 22.067999999999998 |
|
- type: map_at_1000 |
|
value: 22.192 |
|
- type: map_at_3 |
|
value: 18.673000000000002 |
|
- type: map_at_5 |
|
value: 19.968 |
|
- type: mrr_at_1 |
|
value: 17.785999999999998 |
|
- type: mrr_at_10 |
|
value: 24.878 |
|
- type: mrr_at_100 |
|
value: 26.021 |
|
- type: mrr_at_1000 |
|
value: 26.095000000000002 |
|
- type: mrr_at_3 |
|
value: 22.616 |
|
- type: mrr_at_5 |
|
value: 23.785 |
|
- type: ndcg_at_1 |
|
value: 17.785999999999998 |
|
- type: ndcg_at_10 |
|
value: 25.153 |
|
- type: ndcg_at_100 |
|
value: 31.05 |
|
- type: ndcg_at_1000 |
|
value: 34.052 |
|
- type: ndcg_at_3 |
|
value: 21.117 |
|
- type: ndcg_at_5 |
|
value: 23.048 |
|
- type: precision_at_1 |
|
value: 17.785999999999998 |
|
- type: precision_at_10 |
|
value: 4.590000000000001 |
|
- type: precision_at_100 |
|
value: 0.864 |
|
- type: precision_at_1000 |
|
value: 0.125 |
|
- type: precision_at_3 |
|
value: 9.908999999999999 |
|
- type: precision_at_5 |
|
value: 7.313 |
|
- type: recall_at_1 |
|
value: 14.419 |
|
- type: recall_at_10 |
|
value: 34.477999999999994 |
|
- type: recall_at_100 |
|
value: 60.02499999999999 |
|
- type: recall_at_1000 |
|
value: 81.646 |
|
- type: recall_at_3 |
|
value: 23.515 |
|
- type: recall_at_5 |
|
value: 28.266999999999996 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackPhysicsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 26.268 |
|
- type: map_at_10 |
|
value: 35.114000000000004 |
|
- type: map_at_100 |
|
value: 36.212 |
|
- type: map_at_1000 |
|
value: 36.333 |
|
- type: map_at_3 |
|
value: 32.436 |
|
- type: map_at_5 |
|
value: 33.992 |
|
- type: mrr_at_1 |
|
value: 31.761 |
|
- type: mrr_at_10 |
|
value: 40.355999999999995 |
|
- type: mrr_at_100 |
|
value: 41.125 |
|
- type: mrr_at_1000 |
|
value: 41.186 |
|
- type: mrr_at_3 |
|
value: 37.937 |
|
- type: mrr_at_5 |
|
value: 39.463 |
|
- type: ndcg_at_1 |
|
value: 31.761 |
|
- type: ndcg_at_10 |
|
value: 40.422000000000004 |
|
- type: ndcg_at_100 |
|
value: 45.458999999999996 |
|
- type: ndcg_at_1000 |
|
value: 47.951 |
|
- type: ndcg_at_3 |
|
value: 35.972 |
|
- type: ndcg_at_5 |
|
value: 38.272 |
|
- type: precision_at_1 |
|
value: 31.761 |
|
- type: precision_at_10 |
|
value: 7.103 |
|
- type: precision_at_100 |
|
value: 1.133 |
|
- type: precision_at_1000 |
|
value: 0.152 |
|
- type: precision_at_3 |
|
value: 16.779 |
|
- type: precision_at_5 |
|
value: 11.877 |
|
- type: recall_at_1 |
|
value: 26.268 |
|
- type: recall_at_10 |
|
value: 51.053000000000004 |
|
- type: recall_at_100 |
|
value: 72.702 |
|
- type: recall_at_1000 |
|
value: 89.521 |
|
- type: recall_at_3 |
|
value: 38.619 |
|
- type: recall_at_5 |
|
value: 44.671 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackProgrammersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 25.230999999999998 |
|
- type: map_at_10 |
|
value: 34.227000000000004 |
|
- type: map_at_100 |
|
value: 35.370000000000005 |
|
- type: map_at_1000 |
|
value: 35.488 |
|
- type: map_at_3 |
|
value: 31.496000000000002 |
|
- type: map_at_5 |
|
value: 33.034 |
|
- type: mrr_at_1 |
|
value: 30.822 |
|
- type: mrr_at_10 |
|
value: 39.045 |
|
- type: mrr_at_100 |
|
value: 39.809 |
|
- type: mrr_at_1000 |
|
value: 39.873 |
|
- type: mrr_at_3 |
|
value: 36.663000000000004 |
|
- type: mrr_at_5 |
|
value: 37.964 |
|
- type: ndcg_at_1 |
|
value: 30.822 |
|
- type: ndcg_at_10 |
|
value: 39.472 |
|
- type: ndcg_at_100 |
|
value: 44.574999999999996 |
|
- type: ndcg_at_1000 |
|
value: 47.162 |
|
- type: ndcg_at_3 |
|
value: 34.929 |
|
- type: ndcg_at_5 |
|
value: 37.002 |
|
- type: precision_at_1 |
|
value: 30.822 |
|
- type: precision_at_10 |
|
value: 7.055 |
|
- type: precision_at_100 |
|
value: 1.124 |
|
- type: precision_at_1000 |
|
value: 0.152 |
|
- type: precision_at_3 |
|
value: 16.591 |
|
- type: precision_at_5 |
|
value: 11.667 |
|
- type: recall_at_1 |
|
value: 25.230999999999998 |
|
- type: recall_at_10 |
|
value: 50.42100000000001 |
|
- type: recall_at_100 |
|
value: 72.685 |
|
- type: recall_at_1000 |
|
value: 90.469 |
|
- type: recall_at_3 |
|
value: 37.503 |
|
- type: recall_at_5 |
|
value: 43.123 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 24.604166666666664 |
|
- type: map_at_10 |
|
value: 32.427166666666665 |
|
- type: map_at_100 |
|
value: 33.51474999999999 |
|
- type: map_at_1000 |
|
value: 33.6345 |
|
- type: map_at_3 |
|
value: 30.02366666666667 |
|
- type: map_at_5 |
|
value: 31.382333333333328 |
|
- type: mrr_at_1 |
|
value: 29.001166666666666 |
|
- type: mrr_at_10 |
|
value: 36.3315 |
|
- type: mrr_at_100 |
|
value: 37.16683333333333 |
|
- type: mrr_at_1000 |
|
value: 37.23341666666668 |
|
- type: mrr_at_3 |
|
value: 34.19916666666667 |
|
- type: mrr_at_5 |
|
value: 35.40458333333334 |
|
- type: ndcg_at_1 |
|
value: 29.001166666666666 |
|
- type: ndcg_at_10 |
|
value: 37.06883333333334 |
|
- type: ndcg_at_100 |
|
value: 41.95816666666666 |
|
- type: ndcg_at_1000 |
|
value: 44.501583333333336 |
|
- type: ndcg_at_3 |
|
value: 32.973499999999994 |
|
- type: ndcg_at_5 |
|
value: 34.90833333333334 |
|
- type: precision_at_1 |
|
value: 29.001166666666666 |
|
- type: precision_at_10 |
|
value: 6.336 |
|
- type: precision_at_100 |
|
value: 1.0282499999999999 |
|
- type: precision_at_1000 |
|
value: 0.14391666666666664 |
|
- type: precision_at_3 |
|
value: 14.932499999999996 |
|
- type: precision_at_5 |
|
value: 10.50825 |
|
- type: recall_at_1 |
|
value: 24.604166666666664 |
|
- type: recall_at_10 |
|
value: 46.9525 |
|
- type: recall_at_100 |
|
value: 68.67816666666667 |
|
- type: recall_at_1000 |
|
value: 86.59783333333334 |
|
- type: recall_at_3 |
|
value: 35.49783333333333 |
|
- type: recall_at_5 |
|
value: 40.52525000000001 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackStatsRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.559 |
|
- type: map_at_10 |
|
value: 29.023 |
|
- type: map_at_100 |
|
value: 29.818 |
|
- type: map_at_1000 |
|
value: 29.909000000000002 |
|
- type: map_at_3 |
|
value: 27.037 |
|
- type: map_at_5 |
|
value: 28.225 |
|
- type: mrr_at_1 |
|
value: 26.994 |
|
- type: mrr_at_10 |
|
value: 31.962000000000003 |
|
- type: mrr_at_100 |
|
value: 32.726 |
|
- type: mrr_at_1000 |
|
value: 32.800000000000004 |
|
- type: mrr_at_3 |
|
value: 30.266 |
|
- type: mrr_at_5 |
|
value: 31.208999999999996 |
|
- type: ndcg_at_1 |
|
value: 26.994 |
|
- type: ndcg_at_10 |
|
value: 32.53 |
|
- type: ndcg_at_100 |
|
value: 36.758 |
|
- type: ndcg_at_1000 |
|
value: 39.362 |
|
- type: ndcg_at_3 |
|
value: 28.985 |
|
- type: ndcg_at_5 |
|
value: 30.757 |
|
- type: precision_at_1 |
|
value: 26.994 |
|
- type: precision_at_10 |
|
value: 4.968999999999999 |
|
- type: precision_at_100 |
|
value: 0.759 |
|
- type: precision_at_1000 |
|
value: 0.106 |
|
- type: precision_at_3 |
|
value: 12.219 |
|
- type: precision_at_5 |
|
value: 8.527999999999999 |
|
- type: recall_at_1 |
|
value: 23.559 |
|
- type: recall_at_10 |
|
value: 40.585 |
|
- type: recall_at_100 |
|
value: 60.306000000000004 |
|
- type: recall_at_1000 |
|
value: 80.11 |
|
- type: recall_at_3 |
|
value: 30.794 |
|
- type: recall_at_5 |
|
value: 35.186 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackTexRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 16.384999999999998 |
|
- type: map_at_10 |
|
value: 22.142 |
|
- type: map_at_100 |
|
value: 23.057 |
|
- type: map_at_1000 |
|
value: 23.177 |
|
- type: map_at_3 |
|
value: 20.29 |
|
- type: map_at_5 |
|
value: 21.332 |
|
- type: mrr_at_1 |
|
value: 19.89 |
|
- type: mrr_at_10 |
|
value: 25.771 |
|
- type: mrr_at_100 |
|
value: 26.599 |
|
- type: mrr_at_1000 |
|
value: 26.680999999999997 |
|
- type: mrr_at_3 |
|
value: 23.962 |
|
- type: mrr_at_5 |
|
value: 24.934 |
|
- type: ndcg_at_1 |
|
value: 19.89 |
|
- type: ndcg_at_10 |
|
value: 25.97 |
|
- type: ndcg_at_100 |
|
value: 30.605 |
|
- type: ndcg_at_1000 |
|
value: 33.619 |
|
- type: ndcg_at_3 |
|
value: 22.704 |
|
- type: ndcg_at_5 |
|
value: 24.199 |
|
- type: precision_at_1 |
|
value: 19.89 |
|
- type: precision_at_10 |
|
value: 4.553 |
|
- type: precision_at_100 |
|
value: 0.8049999999999999 |
|
- type: precision_at_1000 |
|
value: 0.122 |
|
- type: precision_at_3 |
|
value: 10.541 |
|
- type: precision_at_5 |
|
value: 7.46 |
|
- type: recall_at_1 |
|
value: 16.384999999999998 |
|
- type: recall_at_10 |
|
value: 34.001 |
|
- type: recall_at_100 |
|
value: 55.17100000000001 |
|
- type: recall_at_1000 |
|
value: 77.125 |
|
- type: recall_at_3 |
|
value: 24.618000000000002 |
|
- type: recall_at_5 |
|
value: 28.695999999999998 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackUnixRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.726 |
|
- type: map_at_10 |
|
value: 31.227 |
|
- type: map_at_100 |
|
value: 32.311 |
|
- type: map_at_1000 |
|
value: 32.419 |
|
- type: map_at_3 |
|
value: 28.765 |
|
- type: map_at_5 |
|
value: 30.229 |
|
- type: mrr_at_1 |
|
value: 27.705000000000002 |
|
- type: mrr_at_10 |
|
value: 35.085 |
|
- type: mrr_at_100 |
|
value: 35.931000000000004 |
|
- type: mrr_at_1000 |
|
value: 36 |
|
- type: mrr_at_3 |
|
value: 32.603 |
|
- type: mrr_at_5 |
|
value: 34.117999999999995 |
|
- type: ndcg_at_1 |
|
value: 27.705000000000002 |
|
- type: ndcg_at_10 |
|
value: 35.968 |
|
- type: ndcg_at_100 |
|
value: 41.197 |
|
- type: ndcg_at_1000 |
|
value: 43.76 |
|
- type: ndcg_at_3 |
|
value: 31.304 |
|
- type: ndcg_at_5 |
|
value: 33.661 |
|
- type: precision_at_1 |
|
value: 27.705000000000002 |
|
- type: precision_at_10 |
|
value: 5.942 |
|
- type: precision_at_100 |
|
value: 0.964 |
|
- type: precision_at_1000 |
|
value: 0.13 |
|
- type: precision_at_3 |
|
value: 13.868 |
|
- type: precision_at_5 |
|
value: 9.944 |
|
- type: recall_at_1 |
|
value: 23.726 |
|
- type: recall_at_10 |
|
value: 46.786 |
|
- type: recall_at_100 |
|
value: 70.072 |
|
- type: recall_at_1000 |
|
value: 88.2 |
|
- type: recall_at_3 |
|
value: 33.981 |
|
- type: recall_at_5 |
|
value: 39.893 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWebmastersRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 23.344 |
|
- type: map_at_10 |
|
value: 31.636999999999997 |
|
- type: map_at_100 |
|
value: 33.065 |
|
- type: map_at_1000 |
|
value: 33.300000000000004 |
|
- type: map_at_3 |
|
value: 29.351 |
|
- type: map_at_5 |
|
value: 30.432 |
|
- type: mrr_at_1 |
|
value: 27.866000000000003 |
|
- type: mrr_at_10 |
|
value: 35.587 |
|
- type: mrr_at_100 |
|
value: 36.52 |
|
- type: mrr_at_1000 |
|
value: 36.597 |
|
- type: mrr_at_3 |
|
value: 33.696 |
|
- type: mrr_at_5 |
|
value: 34.713 |
|
- type: ndcg_at_1 |
|
value: 27.866000000000003 |
|
- type: ndcg_at_10 |
|
value: 36.61 |
|
- type: ndcg_at_100 |
|
value: 41.88 |
|
- type: ndcg_at_1000 |
|
value: 45.105000000000004 |
|
- type: ndcg_at_3 |
|
value: 33.038000000000004 |
|
- type: ndcg_at_5 |
|
value: 34.331 |
|
- type: precision_at_1 |
|
value: 27.866000000000003 |
|
- type: precision_at_10 |
|
value: 6.917 |
|
- type: precision_at_100 |
|
value: 1.3599999999999999 |
|
- type: precision_at_1000 |
|
value: 0.233 |
|
- type: precision_at_3 |
|
value: 15.547 |
|
- type: precision_at_5 |
|
value: 10.791 |
|
- type: recall_at_1 |
|
value: 23.344 |
|
- type: recall_at_10 |
|
value: 45.782000000000004 |
|
- type: recall_at_100 |
|
value: 69.503 |
|
- type: recall_at_1000 |
|
value: 90.742 |
|
- type: recall_at_3 |
|
value: 35.160000000000004 |
|
- type: recall_at_5 |
|
value: 39.058 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: BeIR/cqadupstack |
|
name: MTEB CQADupstackWordpressRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 20.776 |
|
- type: map_at_10 |
|
value: 27.285999999999998 |
|
- type: map_at_100 |
|
value: 28.235 |
|
- type: map_at_1000 |
|
value: 28.337 |
|
- type: map_at_3 |
|
value: 25.147000000000002 |
|
- type: map_at_5 |
|
value: 26.401999999999997 |
|
- type: mrr_at_1 |
|
value: 22.921 |
|
- type: mrr_at_10 |
|
value: 29.409999999999997 |
|
- type: mrr_at_100 |
|
value: 30.275000000000002 |
|
- type: mrr_at_1000 |
|
value: 30.354999999999997 |
|
- type: mrr_at_3 |
|
value: 27.418 |
|
- type: mrr_at_5 |
|
value: 28.592000000000002 |
|
- type: ndcg_at_1 |
|
value: 22.921 |
|
- type: ndcg_at_10 |
|
value: 31.239 |
|
- type: ndcg_at_100 |
|
value: 35.965 |
|
- type: ndcg_at_1000 |
|
value: 38.602 |
|
- type: ndcg_at_3 |
|
value: 27.174 |
|
- type: ndcg_at_5 |
|
value: 29.229 |
|
- type: precision_at_1 |
|
value: 22.921 |
|
- type: precision_at_10 |
|
value: 4.806 |
|
- type: precision_at_100 |
|
value: 0.776 |
|
- type: precision_at_1000 |
|
value: 0.11 |
|
- type: precision_at_3 |
|
value: 11.459999999999999 |
|
- type: precision_at_5 |
|
value: 8.022 |
|
- type: recall_at_1 |
|
value: 20.776 |
|
- type: recall_at_10 |
|
value: 41.294 |
|
- type: recall_at_100 |
|
value: 63.111 |
|
- type: recall_at_1000 |
|
value: 82.88600000000001 |
|
- type: recall_at_3 |
|
value: 30.403000000000002 |
|
- type: recall_at_5 |
|
value: 35.455999999999996 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: climate-fever |
|
name: MTEB ClimateFEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 9.376 |
|
- type: map_at_10 |
|
value: 15.926000000000002 |
|
- type: map_at_100 |
|
value: 17.585 |
|
- type: map_at_1000 |
|
value: 17.776 |
|
- type: map_at_3 |
|
value: 13.014000000000001 |
|
- type: map_at_5 |
|
value: 14.417 |
|
- type: mrr_at_1 |
|
value: 20.195 |
|
- type: mrr_at_10 |
|
value: 29.95 |
|
- type: mrr_at_100 |
|
value: 31.052000000000003 |
|
- type: mrr_at_1000 |
|
value: 31.108000000000004 |
|
- type: mrr_at_3 |
|
value: 26.667 |
|
- type: mrr_at_5 |
|
value: 28.458 |
|
- type: ndcg_at_1 |
|
value: 20.195 |
|
- type: ndcg_at_10 |
|
value: 22.871 |
|
- type: ndcg_at_100 |
|
value: 29.921999999999997 |
|
- type: ndcg_at_1000 |
|
value: 33.672999999999995 |
|
- type: ndcg_at_3 |
|
value: 17.782999999999998 |
|
- type: ndcg_at_5 |
|
value: 19.544 |
|
- type: precision_at_1 |
|
value: 20.195 |
|
- type: precision_at_10 |
|
value: 7.394 |
|
- type: precision_at_100 |
|
value: 1.493 |
|
- type: precision_at_1000 |
|
value: 0.218 |
|
- type: precision_at_3 |
|
value: 13.073 |
|
- type: precision_at_5 |
|
value: 10.436 |
|
- type: recall_at_1 |
|
value: 9.376 |
|
- type: recall_at_10 |
|
value: 28.544999999999998 |
|
- type: recall_at_100 |
|
value: 53.147999999999996 |
|
- type: recall_at_1000 |
|
value: 74.62 |
|
- type: recall_at_3 |
|
value: 16.464000000000002 |
|
- type: recall_at_5 |
|
value: 21.004 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: dbpedia-entity |
|
name: MTEB DBPedia |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 8.415000000000001 |
|
- type: map_at_10 |
|
value: 18.738 |
|
- type: map_at_100 |
|
value: 27.291999999999998 |
|
- type: map_at_1000 |
|
value: 28.992 |
|
- type: map_at_3 |
|
value: 13.196 |
|
- type: map_at_5 |
|
value: 15.539 |
|
- type: mrr_at_1 |
|
value: 66.5 |
|
- type: mrr_at_10 |
|
value: 74.518 |
|
- type: mrr_at_100 |
|
value: 74.86 |
|
- type: mrr_at_1000 |
|
value: 74.87 |
|
- type: mrr_at_3 |
|
value: 72.375 |
|
- type: mrr_at_5 |
|
value: 73.86200000000001 |
|
- type: ndcg_at_1 |
|
value: 54.37499999999999 |
|
- type: ndcg_at_10 |
|
value: 41.317 |
|
- type: ndcg_at_100 |
|
value: 45.845 |
|
- type: ndcg_at_1000 |
|
value: 52.92 |
|
- type: ndcg_at_3 |
|
value: 44.983000000000004 |
|
- type: ndcg_at_5 |
|
value: 42.989 |
|
- type: precision_at_1 |
|
value: 66.5 |
|
- type: precision_at_10 |
|
value: 33.6 |
|
- type: precision_at_100 |
|
value: 10.972999999999999 |
|
- type: precision_at_1000 |
|
value: 2.214 |
|
- type: precision_at_3 |
|
value: 48.583 |
|
- type: precision_at_5 |
|
value: 42.15 |
|
- type: recall_at_1 |
|
value: 8.415000000000001 |
|
- type: recall_at_10 |
|
value: 24.953 |
|
- type: recall_at_100 |
|
value: 52.48199999999999 |
|
- type: recall_at_1000 |
|
value: 75.093 |
|
- type: recall_at_3 |
|
value: 14.341000000000001 |
|
- type: recall_at_5 |
|
value: 18.468 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/emotion |
|
name: MTEB EmotionClassification |
|
config: default |
|
split: test |
|
revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37 |
|
metrics: |
|
- type: accuracy |
|
value: 47.06499999999999 |
|
- type: f1 |
|
value: 41.439327599975385 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fever |
|
name: MTEB FEVER |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 66.02 |
|
- type: map_at_10 |
|
value: 76.68599999999999 |
|
- type: map_at_100 |
|
value: 76.959 |
|
- type: map_at_1000 |
|
value: 76.972 |
|
- type: map_at_3 |
|
value: 75.024 |
|
- type: map_at_5 |
|
value: 76.153 |
|
- type: mrr_at_1 |
|
value: 71.197 |
|
- type: mrr_at_10 |
|
value: 81.105 |
|
- type: mrr_at_100 |
|
value: 81.232 |
|
- type: mrr_at_1000 |
|
value: 81.233 |
|
- type: mrr_at_3 |
|
value: 79.758 |
|
- type: mrr_at_5 |
|
value: 80.69 |
|
- type: ndcg_at_1 |
|
value: 71.197 |
|
- type: ndcg_at_10 |
|
value: 81.644 |
|
- type: ndcg_at_100 |
|
value: 82.645 |
|
- type: ndcg_at_1000 |
|
value: 82.879 |
|
- type: ndcg_at_3 |
|
value: 78.792 |
|
- type: ndcg_at_5 |
|
value: 80.528 |
|
- type: precision_at_1 |
|
value: 71.197 |
|
- type: precision_at_10 |
|
value: 10.206999999999999 |
|
- type: precision_at_100 |
|
value: 1.093 |
|
- type: precision_at_1000 |
|
value: 0.11299999999999999 |
|
- type: precision_at_3 |
|
value: 30.868000000000002 |
|
- type: precision_at_5 |
|
value: 19.559 |
|
- type: recall_at_1 |
|
value: 66.02 |
|
- type: recall_at_10 |
|
value: 92.50699999999999 |
|
- type: recall_at_100 |
|
value: 96.497 |
|
- type: recall_at_1000 |
|
value: 97.956 |
|
- type: recall_at_3 |
|
value: 84.866 |
|
- type: recall_at_5 |
|
value: 89.16199999999999 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: fiqa |
|
name: MTEB FiQA2018 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 17.948 |
|
- type: map_at_10 |
|
value: 29.833 |
|
- type: map_at_100 |
|
value: 31.487 |
|
- type: map_at_1000 |
|
value: 31.674000000000003 |
|
- type: map_at_3 |
|
value: 26.029999999999998 |
|
- type: map_at_5 |
|
value: 28.038999999999998 |
|
- type: mrr_at_1 |
|
value: 34.721999999999994 |
|
- type: mrr_at_10 |
|
value: 44.214999999999996 |
|
- type: mrr_at_100 |
|
value: 44.994 |
|
- type: mrr_at_1000 |
|
value: 45.051 |
|
- type: mrr_at_3 |
|
value: 41.667 |
|
- type: mrr_at_5 |
|
value: 43.032 |
|
- type: ndcg_at_1 |
|
value: 34.721999999999994 |
|
- type: ndcg_at_10 |
|
value: 37.434 |
|
- type: ndcg_at_100 |
|
value: 43.702000000000005 |
|
- type: ndcg_at_1000 |
|
value: 46.993 |
|
- type: ndcg_at_3 |
|
value: 33.56 |
|
- type: ndcg_at_5 |
|
value: 34.687 |
|
- type: precision_at_1 |
|
value: 34.721999999999994 |
|
- type: precision_at_10 |
|
value: 10.401 |
|
- type: precision_at_100 |
|
value: 1.7049999999999998 |
|
- type: precision_at_1000 |
|
value: 0.22799999999999998 |
|
- type: precision_at_3 |
|
value: 22.531000000000002 |
|
- type: precision_at_5 |
|
value: 16.42 |
|
- type: recall_at_1 |
|
value: 17.948 |
|
- type: recall_at_10 |
|
value: 45.062999999999995 |
|
- type: recall_at_100 |
|
value: 68.191 |
|
- type: recall_at_1000 |
|
value: 87.954 |
|
- type: recall_at_3 |
|
value: 31.112000000000002 |
|
- type: recall_at_5 |
|
value: 36.823 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: hotpotqa |
|
name: MTEB HotpotQA |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 36.644 |
|
- type: map_at_10 |
|
value: 57.658 |
|
- type: map_at_100 |
|
value: 58.562000000000005 |
|
- type: map_at_1000 |
|
value: 58.62500000000001 |
|
- type: map_at_3 |
|
value: 54.022999999999996 |
|
- type: map_at_5 |
|
value: 56.293000000000006 |
|
- type: mrr_at_1 |
|
value: 73.288 |
|
- type: mrr_at_10 |
|
value: 80.51700000000001 |
|
- type: mrr_at_100 |
|
value: 80.72 |
|
- type: mrr_at_1000 |
|
value: 80.728 |
|
- type: mrr_at_3 |
|
value: 79.33200000000001 |
|
- type: mrr_at_5 |
|
value: 80.085 |
|
- type: ndcg_at_1 |
|
value: 73.288 |
|
- type: ndcg_at_10 |
|
value: 66.61 |
|
- type: ndcg_at_100 |
|
value: 69.723 |
|
- type: ndcg_at_1000 |
|
value: 70.96000000000001 |
|
- type: ndcg_at_3 |
|
value: 61.358999999999995 |
|
- type: ndcg_at_5 |
|
value: 64.277 |
|
- type: precision_at_1 |
|
value: 73.288 |
|
- type: precision_at_10 |
|
value: 14.17 |
|
- type: precision_at_100 |
|
value: 1.659 |
|
- type: precision_at_1000 |
|
value: 0.182 |
|
- type: precision_at_3 |
|
value: 39.487 |
|
- type: precision_at_5 |
|
value: 25.999 |
|
- type: recall_at_1 |
|
value: 36.644 |
|
- type: recall_at_10 |
|
value: 70.851 |
|
- type: recall_at_100 |
|
value: 82.94399999999999 |
|
- type: recall_at_1000 |
|
value: 91.134 |
|
- type: recall_at_3 |
|
value: 59.230000000000004 |
|
- type: recall_at_5 |
|
value: 64.997 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/imdb |
|
name: MTEB ImdbClassification |
|
config: default |
|
split: test |
|
revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7 |
|
metrics: |
|
- type: accuracy |
|
value: 86.00280000000001 |
|
- type: ap |
|
value: 80.46302061021223 |
|
- type: f1 |
|
value: 85.9592921596419 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: msmarco |
|
name: MTEB MSMARCO |
|
config: default |
|
split: dev |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 22.541 |
|
- type: map_at_10 |
|
value: 34.625 |
|
- type: map_at_100 |
|
value: 35.785 |
|
- type: map_at_1000 |
|
value: 35.831 |
|
- type: map_at_3 |
|
value: 30.823 |
|
- type: map_at_5 |
|
value: 32.967999999999996 |
|
- type: mrr_at_1 |
|
value: 23.180999999999997 |
|
- type: mrr_at_10 |
|
value: 35.207 |
|
- type: mrr_at_100 |
|
value: 36.315 |
|
- type: mrr_at_1000 |
|
value: 36.355 |
|
- type: mrr_at_3 |
|
value: 31.483 |
|
- type: mrr_at_5 |
|
value: 33.589999999999996 |
|
- type: ndcg_at_1 |
|
value: 23.195 |
|
- type: ndcg_at_10 |
|
value: 41.461 |
|
- type: ndcg_at_100 |
|
value: 47.032000000000004 |
|
- type: ndcg_at_1000 |
|
value: 48.199999999999996 |
|
- type: ndcg_at_3 |
|
value: 33.702 |
|
- type: ndcg_at_5 |
|
value: 37.522 |
|
- type: precision_at_1 |
|
value: 23.195 |
|
- type: precision_at_10 |
|
value: 6.526999999999999 |
|
- type: precision_at_100 |
|
value: 0.932 |
|
- type: precision_at_1000 |
|
value: 0.10300000000000001 |
|
- type: precision_at_3 |
|
value: 14.308000000000002 |
|
- type: precision_at_5 |
|
value: 10.507 |
|
- type: recall_at_1 |
|
value: 22.541 |
|
- type: recall_at_10 |
|
value: 62.524 |
|
- type: recall_at_100 |
|
value: 88.228 |
|
- type: recall_at_1000 |
|
value: 97.243 |
|
- type: recall_at_3 |
|
value: 41.38 |
|
- type: recall_at_5 |
|
value: 50.55 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_domain |
|
name: MTEB MTOPDomainClassification (en) |
|
config: en |
|
split: test |
|
revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf |
|
metrics: |
|
- type: accuracy |
|
value: 92.69949840401279 |
|
- type: f1 |
|
value: 92.54141471311786 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/mtop_intent |
|
name: MTEB MTOPIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba |
|
metrics: |
|
- type: accuracy |
|
value: 72.56041951664386 |
|
- type: f1 |
|
value: 55.88499977508287 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_intent |
|
name: MTEB MassiveIntentClassification (en) |
|
config: en |
|
split: test |
|
revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7 |
|
metrics: |
|
- type: accuracy |
|
value: 71.62071284465365 |
|
- type: f1 |
|
value: 69.36717546572152 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/amazon_massive_scenario |
|
name: MTEB MassiveScenarioClassification (en) |
|
config: en |
|
split: test |
|
revision: 7d571f92784cd94a019292a1f45445077d0ef634 |
|
metrics: |
|
- type: accuracy |
|
value: 76.35843981170142 |
|
- type: f1 |
|
value: 76.15496453538884 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-p2p |
|
name: MTEB MedrxivClusteringP2P |
|
config: default |
|
split: test |
|
revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73 |
|
metrics: |
|
- type: v_measure |
|
value: 31.33664956793118 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/medrxiv-clustering-s2s |
|
name: MTEB MedrxivClusteringS2S |
|
config: default |
|
split: test |
|
revision: 35191c8c0dca72d8ff3efcd72aa802307d469663 |
|
metrics: |
|
- type: v_measure |
|
value: 27.883839621715524 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/mind_small |
|
name: MTEB MindSmallReranking |
|
config: default |
|
split: test |
|
revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69 |
|
metrics: |
|
- type: map |
|
value: 30.096874986740758 |
|
- type: mrr |
|
value: 30.97300481932132 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nfcorpus |
|
name: MTEB NFCorpus |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 5.4 |
|
- type: map_at_10 |
|
value: 11.852 |
|
- type: map_at_100 |
|
value: 14.758 |
|
- type: map_at_1000 |
|
value: 16.134 |
|
- type: map_at_3 |
|
value: 8.558 |
|
- type: map_at_5 |
|
value: 10.087 |
|
- type: mrr_at_1 |
|
value: 44.272 |
|
- type: mrr_at_10 |
|
value: 52.05800000000001 |
|
- type: mrr_at_100 |
|
value: 52.689 |
|
- type: mrr_at_1000 |
|
value: 52.742999999999995 |
|
- type: mrr_at_3 |
|
value: 50.205999999999996 |
|
- type: mrr_at_5 |
|
value: 51.367 |
|
- type: ndcg_at_1 |
|
value: 42.57 |
|
- type: ndcg_at_10 |
|
value: 32.449 |
|
- type: ndcg_at_100 |
|
value: 29.596 |
|
- type: ndcg_at_1000 |
|
value: 38.351 |
|
- type: ndcg_at_3 |
|
value: 37.044 |
|
- type: ndcg_at_5 |
|
value: 35.275 |
|
- type: precision_at_1 |
|
value: 44.272 |
|
- type: precision_at_10 |
|
value: 23.87 |
|
- type: precision_at_100 |
|
value: 7.625 |
|
- type: precision_at_1000 |
|
value: 2.045 |
|
- type: precision_at_3 |
|
value: 34.365 |
|
- type: precision_at_5 |
|
value: 30.341 |
|
- type: recall_at_1 |
|
value: 5.4 |
|
- type: recall_at_10 |
|
value: 15.943999999999999 |
|
- type: recall_at_100 |
|
value: 29.805 |
|
- type: recall_at_1000 |
|
value: 61.695 |
|
- type: recall_at_3 |
|
value: 9.539 |
|
- type: recall_at_5 |
|
value: 12.127 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: nq |
|
name: MTEB NQ |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 36.047000000000004 |
|
- type: map_at_10 |
|
value: 51.6 |
|
- type: map_at_100 |
|
value: 52.449999999999996 |
|
- type: map_at_1000 |
|
value: 52.476 |
|
- type: map_at_3 |
|
value: 47.452 |
|
- type: map_at_5 |
|
value: 49.964 |
|
- type: mrr_at_1 |
|
value: 40.382 |
|
- type: mrr_at_10 |
|
value: 54.273 |
|
- type: mrr_at_100 |
|
value: 54.859 |
|
- type: mrr_at_1000 |
|
value: 54.876000000000005 |
|
- type: mrr_at_3 |
|
value: 51.014 |
|
- type: mrr_at_5 |
|
value: 52.983999999999995 |
|
- type: ndcg_at_1 |
|
value: 40.353 |
|
- type: ndcg_at_10 |
|
value: 59.11300000000001 |
|
- type: ndcg_at_100 |
|
value: 62.604000000000006 |
|
- type: ndcg_at_1000 |
|
value: 63.187000000000005 |
|
- type: ndcg_at_3 |
|
value: 51.513 |
|
- type: ndcg_at_5 |
|
value: 55.576 |
|
- type: precision_at_1 |
|
value: 40.353 |
|
- type: precision_at_10 |
|
value: 9.418 |
|
- type: precision_at_100 |
|
value: 1.1440000000000001 |
|
- type: precision_at_1000 |
|
value: 0.12 |
|
- type: precision_at_3 |
|
value: 23.078000000000003 |
|
- type: precision_at_5 |
|
value: 16.250999999999998 |
|
- type: recall_at_1 |
|
value: 36.047000000000004 |
|
- type: recall_at_10 |
|
value: 79.22200000000001 |
|
- type: recall_at_100 |
|
value: 94.23 |
|
- type: recall_at_1000 |
|
value: 98.51100000000001 |
|
- type: recall_at_3 |
|
value: 59.678 |
|
- type: recall_at_5 |
|
value: 68.967 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: quora |
|
name: MTEB QuoraRetrieval |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 68.232 |
|
- type: map_at_10 |
|
value: 81.674 |
|
- type: map_at_100 |
|
value: 82.338 |
|
- type: map_at_1000 |
|
value: 82.36099999999999 |
|
- type: map_at_3 |
|
value: 78.833 |
|
- type: map_at_5 |
|
value: 80.58 |
|
- type: mrr_at_1 |
|
value: 78.64 |
|
- type: mrr_at_10 |
|
value: 85.164 |
|
- type: mrr_at_100 |
|
value: 85.317 |
|
- type: mrr_at_1000 |
|
value: 85.319 |
|
- type: mrr_at_3 |
|
value: 84.127 |
|
- type: mrr_at_5 |
|
value: 84.789 |
|
- type: ndcg_at_1 |
|
value: 78.63 |
|
- type: ndcg_at_10 |
|
value: 85.711 |
|
- type: ndcg_at_100 |
|
value: 87.238 |
|
- type: ndcg_at_1000 |
|
value: 87.444 |
|
- type: ndcg_at_3 |
|
value: 82.788 |
|
- type: ndcg_at_5 |
|
value: 84.313 |
|
- type: precision_at_1 |
|
value: 78.63 |
|
- type: precision_at_10 |
|
value: 12.977 |
|
- type: precision_at_100 |
|
value: 1.503 |
|
- type: precision_at_1000 |
|
value: 0.156 |
|
- type: precision_at_3 |
|
value: 36.113 |
|
- type: precision_at_5 |
|
value: 23.71 |
|
- type: recall_at_1 |
|
value: 68.232 |
|
- type: recall_at_10 |
|
value: 93.30199999999999 |
|
- type: recall_at_100 |
|
value: 98.799 |
|
- type: recall_at_1000 |
|
value: 99.885 |
|
- type: recall_at_3 |
|
value: 84.827 |
|
- type: recall_at_5 |
|
value: 89.188 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering |
|
name: MTEB RedditClustering |
|
config: default |
|
split: test |
|
revision: 24640382cdbf8abc73003fb0fa6d111a705499eb |
|
metrics: |
|
- type: v_measure |
|
value: 45.71879170816294 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/reddit-clustering-p2p |
|
name: MTEB RedditClusteringP2P |
|
config: default |
|
split: test |
|
revision: 282350215ef01743dc01b456c7f5241fa8937f16 |
|
metrics: |
|
- type: v_measure |
|
value: 59.65866311751794 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scidocs |
|
name: MTEB SCIDOCS |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 4.218 |
|
- type: map_at_10 |
|
value: 10.337 |
|
- type: map_at_100 |
|
value: 12.131 |
|
- type: map_at_1000 |
|
value: 12.411 |
|
- type: map_at_3 |
|
value: 7.4270000000000005 |
|
- type: map_at_5 |
|
value: 8.913 |
|
- type: mrr_at_1 |
|
value: 20.8 |
|
- type: mrr_at_10 |
|
value: 30.868000000000002 |
|
- type: mrr_at_100 |
|
value: 31.903 |
|
- type: mrr_at_1000 |
|
value: 31.972 |
|
- type: mrr_at_3 |
|
value: 27.367 |
|
- type: mrr_at_5 |
|
value: 29.372 |
|
- type: ndcg_at_1 |
|
value: 20.8 |
|
- type: ndcg_at_10 |
|
value: 17.765 |
|
- type: ndcg_at_100 |
|
value: 24.914 |
|
- type: ndcg_at_1000 |
|
value: 30.206 |
|
- type: ndcg_at_3 |
|
value: 16.64 |
|
- type: ndcg_at_5 |
|
value: 14.712 |
|
- type: precision_at_1 |
|
value: 20.8 |
|
- type: precision_at_10 |
|
value: 9.24 |
|
- type: precision_at_100 |
|
value: 1.9560000000000002 |
|
- type: precision_at_1000 |
|
value: 0.32299999999999995 |
|
- type: precision_at_3 |
|
value: 15.467 |
|
- type: precision_at_5 |
|
value: 12.94 |
|
- type: recall_at_1 |
|
value: 4.218 |
|
- type: recall_at_10 |
|
value: 18.752 |
|
- type: recall_at_100 |
|
value: 39.7 |
|
- type: recall_at_1000 |
|
value: 65.57300000000001 |
|
- type: recall_at_3 |
|
value: 9.428 |
|
- type: recall_at_5 |
|
value: 13.133000000000001 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sickr-sts |
|
name: MTEB SICK-R |
|
config: default |
|
split: test |
|
revision: a6ea5a8cab320b040a23452cc28066d9beae2cee |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 83.04338850207233 |
|
- type: cos_sim_spearman |
|
value: 78.5054651430423 |
|
- type: euclidean_pearson |
|
value: 80.30739451228612 |
|
- type: euclidean_spearman |
|
value: 78.48377464299097 |
|
- type: manhattan_pearson |
|
value: 80.40795049052781 |
|
- type: manhattan_spearman |
|
value: 78.49506205443114 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts12-sts |
|
name: MTEB STS12 |
|
config: default |
|
split: test |
|
revision: a0d554a64d88156834ff5ae9920b964011b16384 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.11596224442962 |
|
- type: cos_sim_spearman |
|
value: 76.20997388935461 |
|
- type: euclidean_pearson |
|
value: 80.56858451349109 |
|
- type: euclidean_spearman |
|
value: 75.92659183871186 |
|
- type: manhattan_pearson |
|
value: 80.60246102203844 |
|
- type: manhattan_spearman |
|
value: 76.03018971432664 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts13-sts |
|
name: MTEB STS13 |
|
config: default |
|
split: test |
|
revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.34691640755737 |
|
- type: cos_sim_spearman |
|
value: 82.4018369631579 |
|
- type: euclidean_pearson |
|
value: 81.87673092245366 |
|
- type: euclidean_spearman |
|
value: 82.3671489960678 |
|
- type: manhattan_pearson |
|
value: 81.88222387719948 |
|
- type: manhattan_spearman |
|
value: 82.3816590344736 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts14-sts |
|
name: MTEB STS14 |
|
config: default |
|
split: test |
|
revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.2836092579524 |
|
- type: cos_sim_spearman |
|
value: 78.99982781772064 |
|
- type: euclidean_pearson |
|
value: 80.5184271010527 |
|
- type: euclidean_spearman |
|
value: 78.89777392101904 |
|
- type: manhattan_pearson |
|
value: 80.53585705018664 |
|
- type: manhattan_spearman |
|
value: 78.92898405472994 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts15-sts |
|
name: MTEB STS15 |
|
config: default |
|
split: test |
|
revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 86.7349907750784 |
|
- type: cos_sim_spearman |
|
value: 87.7611234446225 |
|
- type: euclidean_pearson |
|
value: 86.98759326731624 |
|
- type: euclidean_spearman |
|
value: 87.58321319424618 |
|
- type: manhattan_pearson |
|
value: 87.03483090370842 |
|
- type: manhattan_spearman |
|
value: 87.63278333060288 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts16-sts |
|
name: MTEB STS16 |
|
config: default |
|
split: test |
|
revision: 4d8694f8f0e0100860b497b999b3dbed754a0513 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 81.75873694924825 |
|
- type: cos_sim_spearman |
|
value: 83.80237999094724 |
|
- type: euclidean_pearson |
|
value: 83.55023725861537 |
|
- type: euclidean_spearman |
|
value: 84.12744338577744 |
|
- type: manhattan_pearson |
|
value: 83.58816983036232 |
|
- type: manhattan_spearman |
|
value: 84.18520748676501 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts17-crosslingual-sts |
|
name: MTEB STS17 (en-en) |
|
config: en-en |
|
split: test |
|
revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 87.21630882940174 |
|
- type: cos_sim_spearman |
|
value: 87.72382883437031 |
|
- type: euclidean_pearson |
|
value: 88.69933350930333 |
|
- type: euclidean_spearman |
|
value: 88.24660814383081 |
|
- type: manhattan_pearson |
|
value: 88.77331018833499 |
|
- type: manhattan_spearman |
|
value: 88.26109989380632 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/sts22-crosslingual-sts |
|
name: MTEB STS22 (en) |
|
config: en |
|
split: test |
|
revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 61.11854063060489 |
|
- type: cos_sim_spearman |
|
value: 63.14678634195072 |
|
- type: euclidean_pearson |
|
value: 61.679090067000864 |
|
- type: euclidean_spearman |
|
value: 62.28876589509653 |
|
- type: manhattan_pearson |
|
value: 62.082324165511004 |
|
- type: manhattan_spearman |
|
value: 62.56030932816679 |
|
- task: |
|
type: STS |
|
dataset: |
|
type: mteb/stsbenchmark-sts |
|
name: MTEB STSBenchmark |
|
config: default |
|
split: test |
|
revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831 |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 84.00319882832645 |
|
- type: cos_sim_spearman |
|
value: 85.94529772647257 |
|
- type: euclidean_pearson |
|
value: 85.6661390122756 |
|
- type: euclidean_spearman |
|
value: 85.97747815545827 |
|
- type: manhattan_pearson |
|
value: 85.58422770541893 |
|
- type: manhattan_spearman |
|
value: 85.9237139181532 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/scidocs-reranking |
|
name: MTEB SciDocsRR |
|
config: default |
|
split: test |
|
revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab |
|
metrics: |
|
- type: map |
|
value: 79.16198731863916 |
|
- type: mrr |
|
value: 94.25202702163487 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: scifact |
|
name: MTEB SciFact |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 54.761 |
|
- type: map_at_10 |
|
value: 64.396 |
|
- type: map_at_100 |
|
value: 65.07 |
|
- type: map_at_1000 |
|
value: 65.09899999999999 |
|
- type: map_at_3 |
|
value: 61.846000000000004 |
|
- type: map_at_5 |
|
value: 63.284 |
|
- type: mrr_at_1 |
|
value: 57.667 |
|
- type: mrr_at_10 |
|
value: 65.83099999999999 |
|
- type: mrr_at_100 |
|
value: 66.36800000000001 |
|
- type: mrr_at_1000 |
|
value: 66.39399999999999 |
|
- type: mrr_at_3 |
|
value: 64.056 |
|
- type: mrr_at_5 |
|
value: 65.206 |
|
- type: ndcg_at_1 |
|
value: 57.667 |
|
- type: ndcg_at_10 |
|
value: 68.854 |
|
- type: ndcg_at_100 |
|
value: 71.59100000000001 |
|
- type: ndcg_at_1000 |
|
value: 72.383 |
|
- type: ndcg_at_3 |
|
value: 64.671 |
|
- type: ndcg_at_5 |
|
value: 66.796 |
|
- type: precision_at_1 |
|
value: 57.667 |
|
- type: precision_at_10 |
|
value: 9.167 |
|
- type: precision_at_100 |
|
value: 1.053 |
|
- type: precision_at_1000 |
|
value: 0.11199999999999999 |
|
- type: precision_at_3 |
|
value: 25.444 |
|
- type: precision_at_5 |
|
value: 16.667 |
|
- type: recall_at_1 |
|
value: 54.761 |
|
- type: recall_at_10 |
|
value: 80.9 |
|
- type: recall_at_100 |
|
value: 92.767 |
|
- type: recall_at_1000 |
|
value: 99 |
|
- type: recall_at_3 |
|
value: 69.672 |
|
- type: recall_at_5 |
|
value: 75.083 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/sprintduplicatequestions-pairclassification |
|
name: MTEB SprintDuplicateQuestions |
|
config: default |
|
split: test |
|
revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 99.8079207920792 |
|
- type: cos_sim_ap |
|
value: 94.88470927617445 |
|
- type: cos_sim_f1 |
|
value: 90.08179959100204 |
|
- type: cos_sim_precision |
|
value: 92.15481171548117 |
|
- type: cos_sim_recall |
|
value: 88.1 |
|
- type: dot_accuracy |
|
value: 99.58613861386138 |
|
- type: dot_ap |
|
value: 82.94822578881316 |
|
- type: dot_f1 |
|
value: 77.33333333333333 |
|
- type: dot_precision |
|
value: 79.36842105263158 |
|
- type: dot_recall |
|
value: 75.4 |
|
- type: euclidean_accuracy |
|
value: 99.8069306930693 |
|
- type: euclidean_ap |
|
value: 94.81367858031837 |
|
- type: euclidean_f1 |
|
value: 90.01009081735621 |
|
- type: euclidean_precision |
|
value: 90.83503054989816 |
|
- type: euclidean_recall |
|
value: 89.2 |
|
- type: manhattan_accuracy |
|
value: 99.81188118811882 |
|
- type: manhattan_ap |
|
value: 94.91405337220161 |
|
- type: manhattan_f1 |
|
value: 90.2763561924258 |
|
- type: manhattan_precision |
|
value: 92.45283018867924 |
|
- type: manhattan_recall |
|
value: 88.2 |
|
- type: max_accuracy |
|
value: 99.81188118811882 |
|
- type: max_ap |
|
value: 94.91405337220161 |
|
- type: max_f1 |
|
value: 90.2763561924258 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering |
|
name: MTEB StackExchangeClustering |
|
config: default |
|
split: test |
|
revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259 |
|
metrics: |
|
- type: v_measure |
|
value: 58.511599500053094 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/stackexchange-clustering-p2p |
|
name: MTEB StackExchangeClusteringP2P |
|
config: default |
|
split: test |
|
revision: 815ca46b2622cec33ccafc3735d572c266efdb44 |
|
metrics: |
|
- type: v_measure |
|
value: 31.984728147814707 |
|
- task: |
|
type: Reranking |
|
dataset: |
|
type: mteb/stackoverflowdupquestions-reranking |
|
name: MTEB StackOverflowDupQuestions |
|
config: default |
|
split: test |
|
revision: e185fbe320c72810689fc5848eb6114e1ef5ec69 |
|
metrics: |
|
- type: map |
|
value: 49.93428193939015 |
|
- type: mrr |
|
value: 50.916557911043206 |
|
- task: |
|
type: Summarization |
|
dataset: |
|
type: mteb/summeval |
|
name: MTEB SummEval |
|
config: default |
|
split: test |
|
revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c |
|
metrics: |
|
- type: cos_sim_pearson |
|
value: 31.562500894537145 |
|
- type: cos_sim_spearman |
|
value: 31.162587976726307 |
|
- type: dot_pearson |
|
value: 22.633662187735762 |
|
- type: dot_spearman |
|
value: 22.723000282378962 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: trec-covid |
|
name: MTEB TRECCOVID |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 0.219 |
|
- type: map_at_10 |
|
value: 1.871 |
|
- type: map_at_100 |
|
value: 10.487 |
|
- type: map_at_1000 |
|
value: 25.122 |
|
- type: map_at_3 |
|
value: 0.657 |
|
- type: map_at_5 |
|
value: 1.0699999999999998 |
|
- type: mrr_at_1 |
|
value: 84 |
|
- type: mrr_at_10 |
|
value: 89.567 |
|
- type: mrr_at_100 |
|
value: 89.748 |
|
- type: mrr_at_1000 |
|
value: 89.748 |
|
- type: mrr_at_3 |
|
value: 88.667 |
|
- type: mrr_at_5 |
|
value: 89.567 |
|
- type: ndcg_at_1 |
|
value: 80 |
|
- type: ndcg_at_10 |
|
value: 74.533 |
|
- type: ndcg_at_100 |
|
value: 55.839000000000006 |
|
- type: ndcg_at_1000 |
|
value: 49.748 |
|
- type: ndcg_at_3 |
|
value: 79.53099999999999 |
|
- type: ndcg_at_5 |
|
value: 78.245 |
|
- type: precision_at_1 |
|
value: 84 |
|
- type: precision_at_10 |
|
value: 78.4 |
|
- type: precision_at_100 |
|
value: 56.99999999999999 |
|
- type: precision_at_1000 |
|
value: 21.98 |
|
- type: precision_at_3 |
|
value: 85.333 |
|
- type: precision_at_5 |
|
value: 84.8 |
|
- type: recall_at_1 |
|
value: 0.219 |
|
- type: recall_at_10 |
|
value: 2.02 |
|
- type: recall_at_100 |
|
value: 13.555 |
|
- type: recall_at_1000 |
|
value: 46.739999999999995 |
|
- type: recall_at_3 |
|
value: 0.685 |
|
- type: recall_at_5 |
|
value: 1.13 |
|
- task: |
|
type: Retrieval |
|
dataset: |
|
type: webis-touche2020 |
|
name: MTEB Touche2020 |
|
config: default |
|
split: test |
|
revision: None |
|
metrics: |
|
- type: map_at_1 |
|
value: 3.5029999999999997 |
|
- type: map_at_10 |
|
value: 11.042 |
|
- type: map_at_100 |
|
value: 16.326999999999998 |
|
- type: map_at_1000 |
|
value: 17.836 |
|
- type: map_at_3 |
|
value: 6.174 |
|
- type: map_at_5 |
|
value: 7.979 |
|
- type: mrr_at_1 |
|
value: 42.857 |
|
- type: mrr_at_10 |
|
value: 52.617000000000004 |
|
- type: mrr_at_100 |
|
value: 53.351000000000006 |
|
- type: mrr_at_1000 |
|
value: 53.351000000000006 |
|
- type: mrr_at_3 |
|
value: 46.939 |
|
- type: mrr_at_5 |
|
value: 50.714000000000006 |
|
- type: ndcg_at_1 |
|
value: 38.775999999999996 |
|
- type: ndcg_at_10 |
|
value: 27.125 |
|
- type: ndcg_at_100 |
|
value: 35.845 |
|
- type: ndcg_at_1000 |
|
value: 47.377 |
|
- type: ndcg_at_3 |
|
value: 29.633 |
|
- type: ndcg_at_5 |
|
value: 28.378999999999998 |
|
- type: precision_at_1 |
|
value: 42.857 |
|
- type: precision_at_10 |
|
value: 24.082 |
|
- type: precision_at_100 |
|
value: 6.877999999999999 |
|
- type: precision_at_1000 |
|
value: 1.463 |
|
- type: precision_at_3 |
|
value: 29.932 |
|
- type: precision_at_5 |
|
value: 28.571 |
|
- type: recall_at_1 |
|
value: 3.5029999999999997 |
|
- type: recall_at_10 |
|
value: 17.068 |
|
- type: recall_at_100 |
|
value: 43.361 |
|
- type: recall_at_1000 |
|
value: 78.835 |
|
- type: recall_at_3 |
|
value: 6.821000000000001 |
|
- type: recall_at_5 |
|
value: 10.357 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/toxic_conversations_50k |
|
name: MTEB ToxicConversationsClassification |
|
config: default |
|
split: test |
|
revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c |
|
metrics: |
|
- type: accuracy |
|
value: 71.0954 |
|
- type: ap |
|
value: 14.216844153511959 |
|
- type: f1 |
|
value: 54.63687418565117 |
|
- task: |
|
type: Classification |
|
dataset: |
|
type: mteb/tweet_sentiment_extraction |
|
name: MTEB TweetSentimentExtractionClassification |
|
config: default |
|
split: test |
|
revision: d604517c81ca91fe16a244d1248fc021f9ecee7a |
|
metrics: |
|
- type: accuracy |
|
value: 61.46293152235427 |
|
- type: f1 |
|
value: 61.744177921638645 |
|
- task: |
|
type: Clustering |
|
dataset: |
|
type: mteb/twentynewsgroups-clustering |
|
name: MTEB TwentyNewsgroupsClustering |
|
config: default |
|
split: test |
|
revision: 6125ec4e24fa026cec8a478383ee943acfbd5449 |
|
metrics: |
|
- type: v_measure |
|
value: 41.12708617788644 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twittersemeval2015-pairclassification |
|
name: MTEB TwitterSemEval2015 |
|
config: default |
|
split: test |
|
revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1 |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 85.75430649102938 |
|
- type: cos_sim_ap |
|
value: 73.34252536948081 |
|
- type: cos_sim_f1 |
|
value: 67.53758935173774 |
|
- type: cos_sim_precision |
|
value: 63.3672525439408 |
|
- type: cos_sim_recall |
|
value: 72.29551451187335 |
|
- type: dot_accuracy |
|
value: 81.71305954580676 |
|
- type: dot_ap |
|
value: 59.5532209082386 |
|
- type: dot_f1 |
|
value: 56.18466898954705 |
|
- type: dot_precision |
|
value: 47.830923248053395 |
|
- type: dot_recall |
|
value: 68.07387862796834 |
|
- type: euclidean_accuracy |
|
value: 85.81987244441795 |
|
- type: euclidean_ap |
|
value: 73.34325409809446 |
|
- type: euclidean_f1 |
|
value: 67.83451360417443 |
|
- type: euclidean_precision |
|
value: 64.09955388588871 |
|
- type: euclidean_recall |
|
value: 72.0316622691293 |
|
- type: manhattan_accuracy |
|
value: 85.68277999642368 |
|
- type: manhattan_ap |
|
value: 73.1535450121903 |
|
- type: manhattan_f1 |
|
value: 67.928237896289 |
|
- type: manhattan_precision |
|
value: 63.56945722171113 |
|
- type: manhattan_recall |
|
value: 72.9287598944591 |
|
- type: max_accuracy |
|
value: 85.81987244441795 |
|
- type: max_ap |
|
value: 73.34325409809446 |
|
- type: max_f1 |
|
value: 67.928237896289 |
|
- task: |
|
type: PairClassification |
|
dataset: |
|
type: mteb/twitterurlcorpus-pairclassification |
|
name: MTEB TwitterURLCorpus |
|
config: default |
|
split: test |
|
revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf |
|
metrics: |
|
- type: cos_sim_accuracy |
|
value: 88.90441262079403 |
|
- type: cos_sim_ap |
|
value: 85.79331880741438 |
|
- type: cos_sim_f1 |
|
value: 78.31563529842548 |
|
- type: cos_sim_precision |
|
value: 74.6683424102779 |
|
- type: cos_sim_recall |
|
value: 82.33754234678165 |
|
- type: dot_accuracy |
|
value: 84.89928978926534 |
|
- type: dot_ap |
|
value: 75.25819218316 |
|
- type: dot_f1 |
|
value: 69.88730119720536 |
|
- type: dot_precision |
|
value: 64.23362374959665 |
|
- type: dot_recall |
|
value: 76.63227594702803 |
|
- type: euclidean_accuracy |
|
value: 89.01695967710637 |
|
- type: euclidean_ap |
|
value: 85.98986606038852 |
|
- type: euclidean_f1 |
|
value: 78.5277880014722 |
|
- type: euclidean_precision |
|
value: 75.22211253701876 |
|
- type: euclidean_recall |
|
value: 82.13735756082538 |
|
- type: manhattan_accuracy |
|
value: 88.99561454573679 |
|
- type: manhattan_ap |
|
value: 85.92262421793953 |
|
- type: manhattan_f1 |
|
value: 78.38866094740769 |
|
- type: manhattan_precision |
|
value: 76.02373028505282 |
|
- type: manhattan_recall |
|
value: 80.9054511857099 |
|
- type: max_accuracy |
|
value: 89.01695967710637 |
|
- type: max_ap |
|
value: 85.98986606038852 |
|
- type: max_f1 |
|
value: 78.5277880014722 |
|
language: |
|
- en |
|
license: mit |
|
--- |
|
# # Fast-Inference with Ctranslate2 |
|
Speedup inference while reducing memory by 2x-4x using int8 inference in C++ on CPU or GPU. |
|
|
|
quantized version of [intfloat/e5-small-v2](https://huggingface.co/intfloat/e5-small-v2) |
|
```bash |
|
pip install hf-hub-ctranslate2>=2.0.8 ctranslate2>=3.16.0 |
|
``` |
|
Converted on 2023-06-15 using |
|
``` |
|
ct2-transformers-converter --model intfloat/e5-small-v2 --output_dir ~/tmp-ct2fast-e5-small-v2 --force --copy_files tokenizer.json README.md tokenizer_config.json vocab.txt special_tokens_map.json .gitattributes --quantization int8_float16 --trust_remote_code |
|
``` |
|
|
|
Checkpoint compatible to [ctranslate2>=3.16.0](https://github.com/OpenNMT/CTranslate2) |
|
and [hf-hub-ctranslate2>=2.0.8](https://github.com/michaelfeil/hf-hub-ctranslate2) |
|
- `compute_type=int8_float16` for `device="cuda"` |
|
- `compute_type=int8` for `device="cpu"` |
|
|
|
```python |
|
from hf_hub_ctranslate2 import TranslatorCT2fromHfHub, GeneratorCT2fromHfHub |
|
from transformers import AutoTokenizer |
|
|
|
model_name = "michaelfeil/ct2fast-e5-small-v2" |
|
|
|
# use either TranslatorCT2fromHfHub or GeneratorCT2fromHfHub here, depending on model. |
|
model = GeneratorCT2fromHfHub( |
|
# load in int8 on CUDA |
|
model_name_or_path=model_name, |
|
device="cuda", |
|
compute_type="int8_float16", |
|
# tokenizer=AutoTokenizer.from_pretrained("{ORG}/{NAME}") |
|
) |
|
outputs = model.generate( |
|
text=["def fibonnaci(", "User: How are you doing? Bot:"], |
|
max_length=64, |
|
include_prompt_in_result=False |
|
) |
|
print(outputs) |
|
``` |
|
|
|
# Licence and other remarks: |
|
This is just a quantized version. Licence conditions are intended to be idential to original huggingface repo. |
|
|
|
# Original description |
|
|
|
|
|
# E5-small-v2 |
|
|
|
[Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://arxiv.org/pdf/2212.03533.pdf). |
|
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, Furu Wei, arXiv 2022 |
|
|
|
This model has 12 layers and the embedding size is 384. |
|
|
|
## Usage |
|
|
|
Below is an example to encode queries and passages from the MS-MARCO passage ranking dataset. |
|
|
|
```python |
|
import torch.nn.functional as F |
|
|
|
from torch import Tensor |
|
from transformers import AutoTokenizer, AutoModel |
|
|
|
|
|
def average_pool(last_hidden_states: Tensor, |
|
attention_mask: Tensor) -> Tensor: |
|
last_hidden = last_hidden_states.masked_fill(~attention_mask[..., None].bool(), 0.0) |
|
return last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None] |
|
|
|
|
|
# Each input text should start with "query: " or "passage: ". |
|
# For tasks other than retrieval, you can simply use the "query: " prefix. |
|
input_texts = ['query: how much protein should a female eat', |
|
'query: summit define', |
|
"passage: As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.", |
|
"passage: Definition of summit for English Language Learners. : 1 the highest point of a mountain : the top of a mountain. : 2 the highest level. : 3 a meeting or series of meetings between the leaders of two or more governments."] |
|
|
|
tokenizer = AutoTokenizer.from_pretrained('intfloat/e5-small-v2') |
|
model = AutoModel.from_pretrained('intfloat/e5-small-v2') |
|
|
|
# Tokenize the input texts |
|
batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt') |
|
|
|
outputs = model(**batch_dict) |
|
embeddings = average_pool(outputs.last_hidden_state, batch_dict['attention_mask']) |
|
|
|
# (Optionally) normalize embeddings |
|
embeddings = F.normalize(embeddings, p=2, dim=1) |
|
scores = (embeddings[:2] @ embeddings[2:].T) * 100 |
|
print(scores.tolist()) |
|
``` |
|
|
|
## Training Details |
|
|
|
Please refer to our paper at [https://arxiv.org/pdf/2212.03533.pdf](https://arxiv.org/pdf/2212.03533.pdf). |
|
|
|
## Benchmark Evaluation |
|
|
|
Check out [unilm/e5](https://github.com/microsoft/unilm/tree/master/e5) to reproduce evaluation results |
|
on the [BEIR](https://arxiv.org/abs/2104.08663) and [MTEB benchmark](https://arxiv.org/abs/2210.07316). |
|
|
|
## Citation |
|
|
|
If you find our paper or models helpful, please consider cite as follows: |
|
|
|
``` |
|
@article{wang2022text, |
|
title={Text Embeddings by Weakly-Supervised Contrastive Pre-training}, |
|
author={Wang, Liang and Yang, Nan and Huang, Xiaolong and Jiao, Binxing and Yang, Linjun and Jiang, Daxin and Majumder, Rangan and Wei, Furu}, |
|
journal={arXiv preprint arXiv:2212.03533}, |
|
year={2022} |
|
} |
|
``` |
|
|
|
## Limitations |
|
|
|
This model only works for English texts. Long texts will be truncated to at most 512 tokens. |
