How can I support the max_length=2048
Thanks for new push of v2. Curious how we can enlarge max_length to 2048?
This checkpoint is initialized from BertModel, which uses absolute position embeddings. So without re-training on longer contexts, there is no native way to support max_length=2048.
In practice, you can consider splitting long documents into shorter chunks and average their embeddings.
Yes, I was wondering if there is a model beyond sentence embedding. Any code we can finetune this model?
Any codebase for training dense retrievers can be used to finetune this model, our experiments are based on the code at https://github.com/microsoft/unilm/tree/master/simlm
@intfloat interesting. Thanks for your reply. Reading it now. TBH, the self-supervised methodology is really attractive.
Just to confirm: this model is just bi encoder model, right? I am confused in your code it has bi encoder and cross encoder.
Yes, this model is bi-encoder. The cross encoder is used to generate teacher score for knowledge distillation.