Spaces:
Running
on
CPU Upgrade
Can a private model be added to the leaderboard?
Hi guys,
thank for your great work!
I wonder whether you require that the model must be a public one.
If I want to add results of my private text embedding model, can it work?
If you just want to compare the model yourself, yes you can run the leaderboard locally and add your model (by adding the result files to mteb/results)
If you want it visible for everyone, then we require that the model be either public or accessible via an API. Otherwise, there is no way to verify the results.
If you just want to compare the model yourself, yes you can run the leaderboard locally and add your model (by adding the result files to
mteb/results)If you want it visible for everyone, then we require that the model be either public or accessible via an API. Otherwise, there is no way to verify the results.
Hi guys, can you explain in detail that how to run the leaderboard locally? thank you!
I was puzzled by how to get the final score from several indicators. Is there a processed file that can be used? Is this confidential？
You can check this for running it locally: https://huggingface.co/spaces/mteb/leaderboard/discussions/7
Do you mean the final average score? It's simply an average across all datasets
If you just want to compare the model yourself, yes you can run the leaderboard locally and add your model (by adding the result files to
mteb/results)If you want it visible for everyone, then we require that the model be either public or accessible via an API. Otherwise, there is no way to verify the results.
Hi，if I have an API to access this model, how should I do.
You can open a PR here: https://huggingface.co/datasets/mteb/results with the result files; Please also provide a link to your API