Spaces:
Running
on
CPU Upgrade
Can we publish MTEB results to leaderboard for a subset of tasks only?
Hey MTEB Team & Community,
I'm new to MTEB leaderboard and I tried to submit evaluation results by adding MTEB results in a model-index node of a model card metadata for STS tasks only, and the "mteb" tag. When I pushed this, it broke the STS tab of the leaderboard, thus I reverted.
Is it possible to submit results for a subset of tasks only? If yes, how?
Thanks for your help!
Yes it should be possible - In what way did it break the STS tab?
When clicking on the STS tab, nothing was displayed anymore and I needed to click on "Refresh" to see unsorted results. Also, if I remember well, it was displaying a few metrics of other dataset languages under the "English" tab.
Here is the first commit that was creating the issue, I attempted several changes and reverts afterwards trying to identify the issue. May I ask you if you see anything that seems wrong?
Please don't waste too much time on this issue, since I also realized afterwards my results were inaccurate due to having finetuned the model on one of the MTEB datasets (rookie mistake).
It looks okay to me, feel free to try again and if the leaderboard breaks, we will fix it :)