New SOTA! Apply for refreshing the results

#64
by memray - opened

Hi MTEB team,

We are releasing an embedding model SFR-Embedding-Mistral achieving SOTA on the MTEB benchmark with an average score of 67.56.
Could you please help refresh the leaderboard? Thank you!

Best,
Rui

Also I wonder how to add other meta information such as dimension and length?

  • Model Size: 14.22
  • Embedding Dimensions: 4096
  • Sequence Length: 32768

Thank you!

Massive Text Embedding Benchmark org

Hey it will automatically add that metadata once you upload the actual model - do you plan on uploading it?

@Muennighoff yes, we are working on the model uploading. But I don't think it will be available very soon. Would you help us update the leaderboard first? Thank you!

cc: @syavuz @yliu279

Massive Text Embedding Benchmark org

Sounds great & congrats on the performance. If you're not opposed, I recommend we update the LB once the model is released, as it will result in a much better community response.

Currently, users will not be able to do anything once they click on your model. Thus, I think it is better to update the LB once there's a model/api + usage instructions 🙌

I agree as well with @Muennighoff , it's weird to have models that are not released in the leaderboard.

Hi @Muennighoff @eek

Our model is ready on the Hub. Could you please update the LB? Thanks!

cc @syavuz @memray

Massive Text Embedding Benchmark org

Done, congrats!

Hi @Muennighoff ,

Could you please refresh the LB? We updated the model from 32-bit to 16-bit, resulting in a change in the model size. Thank you!

Massive Text Embedding Benchmark org

Done :)

Sign up or log in to comment