@inproceedings{
jiang2024repurposing,
title={Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe},
author={Albert Q. Jiang and Alicja Ziarko and Bartosz Piotrowski and Wenda Li and Mateja Jamnik and Piotr Mi{\l}o{\'s}},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=kVL5rvkqGG}
}
Text embeddings are essential for tasks such as document retrieval, clustering, and semantic similarity assessment. In this paper, we study how to contrastively train text embedding models in a compute-optimal fashion, given a suite of pretrained decoder-only language models. Our innovation is an algorithm that produces optimal configurations of model sizes, data quantities, and fine-tuning methods for text-embedding models at different computational budget levels. The resulting recipe, which we obtain through extensive experiments, can be used by practitioners to make informed design choices for their embedding models. Specifically, our findings suggest that full fine-tuning and Low-Rank Adaptation fine-tuning produce optimal models at lower and higher computational budgets respectively.
Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe
Albert Q. Jiang Alicja Ziarko Bartosz Piotrowski Wenda Li Mateja Jamnik Piotr Miłoś