<story><title>Are we at peak vector database?</title><url>https://softwaredoug.com/blog/2024/01/24/are-we-at-peak-vector-db</url></story><parent_chain><item><author>reissbaker</author><text>IMO we are well past peak cosine-similarity-search as a service. Most people I talk to in the space don&amp;#x27;t bother using specialized vector DBs for that.&lt;p&gt;I think there&amp;#x27;s space for a much more interesting product that is longer-lived (since it&amp;#x27;s harder to implement than just cosine-similarity-search on vectors), which is:&lt;p&gt;1. Fine-tuning OSS embedding models on your real-world query patterns&lt;p&gt;2. Storing and recomputing embeddings for your data as you update the fine-tuned models.&lt;p&gt;MTEB averages are fine, but hardly anyone uses the average result: most use cases are specialized (i.e. classification vs clustering vs retrieval). The best models try to be decent at all of those, but I&amp;#x27;d bet that finetuning on a specific use case would beat a general-purpose model, especially on your own dataset (your retrieval is probably meaningfully different than someone else&amp;#x27;s: code retrieval vs document Q&amp;amp;A, for example). And your queries are usually specialized! People using embeddings for RAG are generally not also trying to use the same embeddings for clustering or classification; and the reverse is true too (your recommendation system is likely different than your search system).&lt;p&gt;And if you&amp;#x27;re fine-tuning new models regularly, you need storage + management, since you&amp;#x27;ll need to recompute the embeddings every time you deploy a new model.&lt;p&gt;I would pay for a service that made (1) and (2) easy.</text></item></parent_chain><comment><author>peterstjohn</author><text>I no longer work there, but Lucidworks has had embedding training as a first-class feature in Fusion since January 2020 (I know because I wrapped up adding it just as COVID became a thing). We definitely saw that even with just slightly out-of-band use of language - e.g. in e-commerce, things like &amp;quot;RD TSHRT XS&amp;quot;, embedding search with open (and closed) models would fall below bog-standard* BM25 lexical search. Once you trained a model, performance would kick up above lexical search…and if you combined lexical _and_ vector search, things were great.&lt;p&gt;Also, a member on our team developed an amazing RNN-based model that still today beats the pants off most embedding models when it comes to speed, and is no slouch on CPU either…&lt;p&gt;(* I&amp;#x27;m being harsh on BM25 - it is a baseline that people often forget in vector search, but it can be a tough one to beat at times)</text></comment>
<story><title>Are we at peak vector database?</title><url>https://softwaredoug.com/blog/2024/01/24/are-we-at-peak-vector-db</url></story><parent_chain><item><author>reissbaker</author><text>IMO we are well past peak cosine-similarity-search as a service. Most people I talk to in the space don&amp;#x27;t bother using specialized vector DBs for that.&lt;p&gt;I think there&amp;#x27;s space for a much more interesting product that is longer-lived (since it&amp;#x27;s harder to implement than just cosine-similarity-search on vectors), which is:&lt;p&gt;1. Fine-tuning OSS embedding models on your real-world query patterns&lt;p&gt;2. Storing and recomputing embeddings for your data as you update the fine-tuned models.&lt;p&gt;MTEB averages are fine, but hardly anyone uses the average result: most use cases are specialized (i.e. classification vs clustering vs retrieval). The best models try to be decent at all of those, but I&amp;#x27;d bet that finetuning on a specific use case would beat a general-purpose model, especially on your own dataset (your retrieval is probably meaningfully different than someone else&amp;#x27;s: code retrieval vs document Q&amp;amp;A, for example). And your queries are usually specialized! People using embeddings for RAG are generally not also trying to use the same embeddings for clustering or classification; and the reverse is true too (your recommendation system is likely different than your search system).&lt;p&gt;And if you&amp;#x27;re fine-tuning new models regularly, you need storage + management, since you&amp;#x27;ll need to recompute the embeddings every time you deploy a new model.&lt;p&gt;I would pay for a service that made (1) and (2) easy.</text></item></parent_chain><comment><author>jn2clark</author><text>We (Marqo) are doing a lot on 1 and 2. There is a huge amount to be done on the ML side of vector search and we are investing heavily in it. I think it has not quite sunk in that vector search systems are ML systems and everything that comes with that. I would love to chat about 1 and 2 so feel free to email me (email is in my profile).</text></comment>