Spaces:
Running
on
CPU Upgrade
Which retrieval task is missing?
Hi!
I ran python scripts/run_mteb_english.py for Bulbasaur and after I hit Refresh on the leaderboard, I miss the stats for both Reranking and Retrieval.
I know why Reranking is not shown (that's because I can't run MindSmallReranking), but I can't figure out why Retrieval is not shown.
To my knowledge all Retrieval evals ran successfully and I even compared task by task (the task name from the list compared with the generated file names).
Which Retrieval eval is missing?
Also, could you please refresh the leaderboard?
Thank you!
You can go to the retrieval tab to see which column might be missing. I've restarted the Leaderboard as well.
- Tom Aarsen
Thank you very much for the super quick refresh! :)
Before the server refresh, I can only see my model in the general page. But that's not important now. :)
After the server refresh, I can see that it doesn't have scores for the CQADupstackRetrieval task.
But that task is not listed in the run_mteb_english.py script, which is the script to run for our models to appear in the leaderboard according to adding_a_model.
I'll run that task and update the stats for Bulbasaur.
Thank you very much for your help! 🤗
Hmm, I think CQADupstackRetrieval stands for the 12 tasks from here: https://github.com/embeddings-benchmark/mteb/blob/main/scripts/run_mteb_english.py#L60-L71 and I have them all.