Even if augmenting LLMs with retrieval is an essential step for the current generation of LLMs [15, 27] it also comes with a cost. Traditional retrieval methods as TF-IDF or BM-25 [29] are only able to retrieve documents with keyword overlap and suffer from lexical gap [4]. In order to try to solve this issue, many pre-trained Transformer encoder based dense models have been proposed [9, 28, 17, 11]. Trained neural models have shown good performance over a variety of retrieval benchmarks but they still struggle in the zero-shot setup for new domains [33]. The quality of the retrieval engine is essential for retrieval-augmented models as this will set the upper bound of the model performance. Moreover, the usage of a retrieval engine, especially when the target document index is huge, can significantly increase the latency of the model and hurt real time applications user experience [3].