Remove ` (default)` from MTEB scores caused by an MTEB bug
Browse filesHello!
## Pull Request overview
* Remove ` (default)` from MTEB scores
## Details
Recently, a bug on MTEB caused the dataset names to be updated from e.g. "ArguAna" to include the config (usually "default", but sometimes a language) in the dataset name. This resulted in a discrepancy between the expected dataset names and the true dataset names in e.g. this model card. This PR should fix it. I've verified that it should work by copying this model card to a private repository and checking that MTEB correctly parses the updated one.
Apologies for the inconvenience. Once we've successfully put this model on MTEB, I'd be glad to help announce the model release on LI/X.
- Tom Aarsen
README.md
CHANGED
@@ -14,7 +14,7 @@ model-index:
|
|
14 |
results:
|
15 |
- dataset:
|
16 |
config: default
|
17 |
-
name: MTEB ArguAna
|
18 |
revision: c22ab2a51041ffd869aaddef7af8d8215647e41a
|
19 |
split: test
|
20 |
type: mteb/arguana
|
@@ -305,7 +305,7 @@ model-index:
|
|
305 |
type: Retrieval
|
306 |
- dataset:
|
307 |
config: default
|
308 |
-
name: MTEB CQADupstackAndroidRetrieval
|
309 |
revision: f46a197baaae43b4f621051089b82a364682dfeb
|
310 |
split: test
|
311 |
type: mteb/cqadupstack-android
|
@@ -596,7 +596,7 @@ model-index:
|
|
596 |
type: Retrieval
|
597 |
- dataset:
|
598 |
config: default
|
599 |
-
name: MTEB CQADupstackEnglishRetrieval
|
600 |
revision: ad9991cb51e31e31e430383c75ffb2885547b5f0
|
601 |
split: test
|
602 |
type: mteb/cqadupstack-english
|
@@ -887,7 +887,7 @@ model-index:
|
|
887 |
type: Retrieval
|
888 |
- dataset:
|
889 |
config: default
|
890 |
-
name: MTEB CQADupstackGamingRetrieval
|
891 |
revision: 4885aa143210c98657558c04aaf3dc47cfb54340
|
892 |
split: test
|
893 |
type: mteb/cqadupstack-gaming
|
@@ -1178,7 +1178,7 @@ model-index:
|
|
1178 |
type: Retrieval
|
1179 |
- dataset:
|
1180 |
config: default
|
1181 |
-
name: MTEB CQADupstackGisRetrieval
|
1182 |
revision: 5003b3064772da1887988e05400cf3806fe491f2
|
1183 |
split: test
|
1184 |
type: mteb/cqadupstack-gis
|
@@ -1469,7 +1469,7 @@ model-index:
|
|
1469 |
type: Retrieval
|
1470 |
- dataset:
|
1471 |
config: default
|
1472 |
-
name: MTEB CQADupstackMathematicaRetrieval
|
1473 |
revision: 90fceea13679c63fe563ded68f3b6f06e50061de
|
1474 |
split: test
|
1475 |
type: mteb/cqadupstack-mathematica
|
@@ -1760,7 +1760,7 @@ model-index:
|
|
1760 |
type: Retrieval
|
1761 |
- dataset:
|
1762 |
config: default
|
1763 |
-
name: MTEB CQADupstackPhysicsRetrieval
|
1764 |
revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4
|
1765 |
split: test
|
1766 |
type: mteb/cqadupstack-physics
|
@@ -2051,7 +2051,7 @@ model-index:
|
|
2051 |
type: Retrieval
|
2052 |
- dataset:
|
2053 |
config: default
|
2054 |
-
name: MTEB CQADupstackProgrammersRetrieval
|
2055 |
revision: 6184bc1440d2dbc7612be22b50686b8826d22b32
|
2056 |
split: test
|
2057 |
type: mteb/cqadupstack-programmers
|
@@ -2342,7 +2342,7 @@ model-index:
|
|
2342 |
type: Retrieval
|
2343 |
- dataset:
|
2344 |
config: default
|
2345 |
-
name: MTEB CQADupstackRetrieval
|
2346 |
revision: CQADupstackRetrieval is a combined dataset
|
2347 |
split: test
|
2348 |
type: mteb/cqadupstack
|
@@ -2355,7 +2355,7 @@ model-index:
|
|
2355 |
type: Retrieval
|
2356 |
- dataset:
|
2357 |
config: default
|
2358 |
-
name: MTEB CQADupstackStatsRetrieval
|
2359 |
revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a
|
2360 |
split: test
|
2361 |
type: mteb/cqadupstack-stats
|
@@ -2646,7 +2646,7 @@ model-index:
|
|
2646 |
type: Retrieval
|
2647 |
- dataset:
|
2648 |
config: default
|
2649 |
-
name: MTEB CQADupstackTexRetrieval
|
2650 |
revision: 46989137a86843e03a6195de44b09deda022eec7
|
2651 |
split: test
|
2652 |
type: mteb/cqadupstack-tex
|
@@ -2937,7 +2937,7 @@ model-index:
|
|
2937 |
type: Retrieval
|
2938 |
- dataset:
|
2939 |
config: default
|
2940 |
-
name: MTEB CQADupstackUnixRetrieval
|
2941 |
revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53
|
2942 |
split: test
|
2943 |
type: mteb/cqadupstack-unix
|
@@ -3228,7 +3228,7 @@ model-index:
|
|
3228 |
type: Retrieval
|
3229 |
- dataset:
|
3230 |
config: default
|
3231 |
-
name: MTEB CQADupstackWebmastersRetrieval
|
3232 |
revision: 160c094312a0e1facb97e55eeddb698c0abe3571
|
3233 |
split: test
|
3234 |
type: mteb/cqadupstack-webmasters
|
@@ -3519,7 +3519,7 @@ model-index:
|
|
3519 |
type: Retrieval
|
3520 |
- dataset:
|
3521 |
config: default
|
3522 |
-
name: MTEB CQADupstackWordpressRetrieval
|
3523 |
revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4
|
3524 |
split: test
|
3525 |
type: mteb/cqadupstack-wordpress
|
@@ -3810,7 +3810,7 @@ model-index:
|
|
3810 |
type: Retrieval
|
3811 |
- dataset:
|
3812 |
config: default
|
3813 |
-
name: MTEB ClimateFEVER
|
3814 |
revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380
|
3815 |
split: test
|
3816 |
type: mteb/climate-fever
|
@@ -4101,7 +4101,7 @@ model-index:
|
|
4101 |
type: Retrieval
|
4102 |
- dataset:
|
4103 |
config: default
|
4104 |
-
name: MTEB DBPedia
|
4105 |
revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659
|
4106 |
split: test
|
4107 |
type: mteb/dbpedia
|
@@ -4392,7 +4392,7 @@ model-index:
|
|
4392 |
type: Retrieval
|
4393 |
- dataset:
|
4394 |
config: default
|
4395 |
-
name: MTEB FEVER
|
4396 |
revision: bea83ef9e8fb933d90a2f1d5515737465d613e12
|
4397 |
split: test
|
4398 |
type: mteb/fever
|
@@ -4683,7 +4683,7 @@ model-index:
|
|
4683 |
type: Retrieval
|
4684 |
- dataset:
|
4685 |
config: default
|
4686 |
-
name: MTEB FiQA2018
|
4687 |
revision: 27a168819829fe9bcd655c2df245fb19452e8e06
|
4688 |
split: test
|
4689 |
type: mteb/fiqa
|
@@ -4974,7 +4974,7 @@ model-index:
|
|
4974 |
type: Retrieval
|
4975 |
- dataset:
|
4976 |
config: default
|
4977 |
-
name: MTEB HotpotQA
|
4978 |
revision: ab518f4d6fcca38d87c25209f94beba119d02014
|
4979 |
split: test
|
4980 |
type: mteb/hotpotqa
|
@@ -5265,7 +5265,7 @@ model-index:
|
|
5265 |
type: Retrieval
|
5266 |
- dataset:
|
5267 |
config: default
|
5268 |
-
name: MTEB MSMARCO
|
5269 |
revision: c5a29a104738b98a9e76336939199e264163d4a0
|
5270 |
split: dev
|
5271 |
type: mteb/msmarco
|
@@ -5556,7 +5556,7 @@ model-index:
|
|
5556 |
type: Retrieval
|
5557 |
- dataset:
|
5558 |
config: default
|
5559 |
-
name: MTEB NFCorpus
|
5560 |
revision: ec0fa4fe99da2ff19ca1214b7966684033a58814
|
5561 |
split: test
|
5562 |
type: mteb/nfcorpus
|
@@ -5847,7 +5847,7 @@ model-index:
|
|
5847 |
type: Retrieval
|
5848 |
- dataset:
|
5849 |
config: default
|
5850 |
-
name: MTEB NQ
|
5851 |
revision: b774495ed302d8c44a3a7ea25c90dbce03968f31
|
5852 |
split: test
|
5853 |
type: mteb/nq
|
@@ -6138,7 +6138,7 @@ model-index:
|
|
6138 |
type: Retrieval
|
6139 |
- dataset:
|
6140 |
config: default
|
6141 |
-
name: MTEB QuoraRetrieval
|
6142 |
revision: e4e08e0b7dbe3c8700f0daef558ff32256715259
|
6143 |
split: test
|
6144 |
type: mteb/quora
|
@@ -6429,7 +6429,7 @@ model-index:
|
|
6429 |
type: Retrieval
|
6430 |
- dataset:
|
6431 |
config: default
|
6432 |
-
name: MTEB SCIDOCS
|
6433 |
revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88
|
6434 |
split: test
|
6435 |
type: mteb/scidocs
|
@@ -6720,7 +6720,7 @@ model-index:
|
|
6720 |
type: Retrieval
|
6721 |
- dataset:
|
6722 |
config: default
|
6723 |
-
name: MTEB SciFact
|
6724 |
revision: 0228b52cf27578f30900b9e5271d331663a030d7
|
6725 |
split: test
|
6726 |
type: mteb/scifact
|
@@ -7011,7 +7011,7 @@ model-index:
|
|
7011 |
type: Retrieval
|
7012 |
- dataset:
|
7013 |
config: default
|
7014 |
-
name: MTEB TRECCOVID
|
7015 |
revision: bb9466bac8153a0349341eb1b22e06409e78ef4e
|
7016 |
split: test
|
7017 |
type: mteb/trec-covid
|
@@ -7302,7 +7302,7 @@ model-index:
|
|
7302 |
type: Retrieval
|
7303 |
- dataset:
|
7304 |
config: default
|
7305 |
-
name: MTEB Touche2020
|
7306 |
revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f
|
7307 |
split: test
|
7308 |
type: mteb/touche2020
|
|
|
14 |
results:
|
15 |
- dataset:
|
16 |
config: default
|
17 |
+
name: MTEB ArguAna
|
18 |
revision: c22ab2a51041ffd869aaddef7af8d8215647e41a
|
19 |
split: test
|
20 |
type: mteb/arguana
|
|
|
305 |
type: Retrieval
|
306 |
- dataset:
|
307 |
config: default
|
308 |
+
name: MTEB CQADupstackAndroidRetrieval
|
309 |
revision: f46a197baaae43b4f621051089b82a364682dfeb
|
310 |
split: test
|
311 |
type: mteb/cqadupstack-android
|
|
|
596 |
type: Retrieval
|
597 |
- dataset:
|
598 |
config: default
|
599 |
+
name: MTEB CQADupstackEnglishRetrieval
|
600 |
revision: ad9991cb51e31e31e430383c75ffb2885547b5f0
|
601 |
split: test
|
602 |
type: mteb/cqadupstack-english
|
|
|
887 |
type: Retrieval
|
888 |
- dataset:
|
889 |
config: default
|
890 |
+
name: MTEB CQADupstackGamingRetrieval
|
891 |
revision: 4885aa143210c98657558c04aaf3dc47cfb54340
|
892 |
split: test
|
893 |
type: mteb/cqadupstack-gaming
|
|
|
1178 |
type: Retrieval
|
1179 |
- dataset:
|
1180 |
config: default
|
1181 |
+
name: MTEB CQADupstackGisRetrieval
|
1182 |
revision: 5003b3064772da1887988e05400cf3806fe491f2
|
1183 |
split: test
|
1184 |
type: mteb/cqadupstack-gis
|
|
|
1469 |
type: Retrieval
|
1470 |
- dataset:
|
1471 |
config: default
|
1472 |
+
name: MTEB CQADupstackMathematicaRetrieval
|
1473 |
revision: 90fceea13679c63fe563ded68f3b6f06e50061de
|
1474 |
split: test
|
1475 |
type: mteb/cqadupstack-mathematica
|
|
|
1760 |
type: Retrieval
|
1761 |
- dataset:
|
1762 |
config: default
|
1763 |
+
name: MTEB CQADupstackPhysicsRetrieval
|
1764 |
revision: 79531abbd1fb92d06c6d6315a0cbbbf5bb247ea4
|
1765 |
split: test
|
1766 |
type: mteb/cqadupstack-physics
|
|
|
2051 |
type: Retrieval
|
2052 |
- dataset:
|
2053 |
config: default
|
2054 |
+
name: MTEB CQADupstackProgrammersRetrieval
|
2055 |
revision: 6184bc1440d2dbc7612be22b50686b8826d22b32
|
2056 |
split: test
|
2057 |
type: mteb/cqadupstack-programmers
|
|
|
2342 |
type: Retrieval
|
2343 |
- dataset:
|
2344 |
config: default
|
2345 |
+
name: MTEB CQADupstackRetrieval
|
2346 |
revision: CQADupstackRetrieval is a combined dataset
|
2347 |
split: test
|
2348 |
type: mteb/cqadupstack
|
|
|
2355 |
type: Retrieval
|
2356 |
- dataset:
|
2357 |
config: default
|
2358 |
+
name: MTEB CQADupstackStatsRetrieval
|
2359 |
revision: 65ac3a16b8e91f9cee4c9828cc7c335575432a2a
|
2360 |
split: test
|
2361 |
type: mteb/cqadupstack-stats
|
|
|
2646 |
type: Retrieval
|
2647 |
- dataset:
|
2648 |
config: default
|
2649 |
+
name: MTEB CQADupstackTexRetrieval
|
2650 |
revision: 46989137a86843e03a6195de44b09deda022eec7
|
2651 |
split: test
|
2652 |
type: mteb/cqadupstack-tex
|
|
|
2937 |
type: Retrieval
|
2938 |
- dataset:
|
2939 |
config: default
|
2940 |
+
name: MTEB CQADupstackUnixRetrieval
|
2941 |
revision: 6c6430d3a6d36f8d2a829195bc5dc94d7e063e53
|
2942 |
split: test
|
2943 |
type: mteb/cqadupstack-unix
|
|
|
3228 |
type: Retrieval
|
3229 |
- dataset:
|
3230 |
config: default
|
3231 |
+
name: MTEB CQADupstackWebmastersRetrieval
|
3232 |
revision: 160c094312a0e1facb97e55eeddb698c0abe3571
|
3233 |
split: test
|
3234 |
type: mteb/cqadupstack-webmasters
|
|
|
3519 |
type: Retrieval
|
3520 |
- dataset:
|
3521 |
config: default
|
3522 |
+
name: MTEB CQADupstackWordpressRetrieval
|
3523 |
revision: 4ffe81d471b1924886b33c7567bfb200e9eec5c4
|
3524 |
split: test
|
3525 |
type: mteb/cqadupstack-wordpress
|
|
|
3810 |
type: Retrieval
|
3811 |
- dataset:
|
3812 |
config: default
|
3813 |
+
name: MTEB ClimateFEVER
|
3814 |
revision: 47f2ac6acb640fc46020b02a5b59fdda04d39380
|
3815 |
split: test
|
3816 |
type: mteb/climate-fever
|
|
|
4101 |
type: Retrieval
|
4102 |
- dataset:
|
4103 |
config: default
|
4104 |
+
name: MTEB DBPedia
|
4105 |
revision: c0f706b76e590d620bd6618b3ca8efdd34e2d659
|
4106 |
split: test
|
4107 |
type: mteb/dbpedia
|
|
|
4392 |
type: Retrieval
|
4393 |
- dataset:
|
4394 |
config: default
|
4395 |
+
name: MTEB FEVER
|
4396 |
revision: bea83ef9e8fb933d90a2f1d5515737465d613e12
|
4397 |
split: test
|
4398 |
type: mteb/fever
|
|
|
4683 |
type: Retrieval
|
4684 |
- dataset:
|
4685 |
config: default
|
4686 |
+
name: MTEB FiQA2018
|
4687 |
revision: 27a168819829fe9bcd655c2df245fb19452e8e06
|
4688 |
split: test
|
4689 |
type: mteb/fiqa
|
|
|
4974 |
type: Retrieval
|
4975 |
- dataset:
|
4976 |
config: default
|
4977 |
+
name: MTEB HotpotQA
|
4978 |
revision: ab518f4d6fcca38d87c25209f94beba119d02014
|
4979 |
split: test
|
4980 |
type: mteb/hotpotqa
|
|
|
5265 |
type: Retrieval
|
5266 |
- dataset:
|
5267 |
config: default
|
5268 |
+
name: MTEB MSMARCO
|
5269 |
revision: c5a29a104738b98a9e76336939199e264163d4a0
|
5270 |
split: dev
|
5271 |
type: mteb/msmarco
|
|
|
5556 |
type: Retrieval
|
5557 |
- dataset:
|
5558 |
config: default
|
5559 |
+
name: MTEB NFCorpus
|
5560 |
revision: ec0fa4fe99da2ff19ca1214b7966684033a58814
|
5561 |
split: test
|
5562 |
type: mteb/nfcorpus
|
|
|
5847 |
type: Retrieval
|
5848 |
- dataset:
|
5849 |
config: default
|
5850 |
+
name: MTEB NQ
|
5851 |
revision: b774495ed302d8c44a3a7ea25c90dbce03968f31
|
5852 |
split: test
|
5853 |
type: mteb/nq
|
|
|
6138 |
type: Retrieval
|
6139 |
- dataset:
|
6140 |
config: default
|
6141 |
+
name: MTEB QuoraRetrieval
|
6142 |
revision: e4e08e0b7dbe3c8700f0daef558ff32256715259
|
6143 |
split: test
|
6144 |
type: mteb/quora
|
|
|
6429 |
type: Retrieval
|
6430 |
- dataset:
|
6431 |
config: default
|
6432 |
+
name: MTEB SCIDOCS
|
6433 |
revision: f8c2fcf00f625baaa80f62ec5bd9e1fff3b8ae88
|
6434 |
split: test
|
6435 |
type: mteb/scidocs
|
|
|
6720 |
type: Retrieval
|
6721 |
- dataset:
|
6722 |
config: default
|
6723 |
+
name: MTEB SciFact
|
6724 |
revision: 0228b52cf27578f30900b9e5271d331663a030d7
|
6725 |
split: test
|
6726 |
type: mteb/scifact
|
|
|
7011 |
type: Retrieval
|
7012 |
- dataset:
|
7013 |
config: default
|
7014 |
+
name: MTEB TRECCOVID
|
7015 |
revision: bb9466bac8153a0349341eb1b22e06409e78ef4e
|
7016 |
split: test
|
7017 |
type: mteb/trec-covid
|
|
|
7302 |
type: Retrieval
|
7303 |
- dataset:
|
7304 |
config: default
|
7305 |
+
name: MTEB Touche2020
|
7306 |
revision: a34f9a33db75fa0cbb21bb5cfc3dae8dc8bec93f
|
7307 |
split: test
|
7308 |
type: mteb/touche2020
|