<story><title>LLMs, RAG, and the missing storage layer for AI</title><url>https://blog.lancedb.com/llms-rag-the-missing-storage-layer-for-ai-28ded35fa984</url></story><parent_chain><item><author>panarky</author><text>The first unstated assumption is that similar vectors are relevant documents, and for many use cases that&amp;#x27;s just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM&amp;#x27;s context, and half or more of them aren&amp;#x27;t relevant, does this make the LLM&amp;#x27;s response more or less relevant?&lt;p&gt;The second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that&amp;#x27;s not true either. If you retrieve the top K vectors according to the vector index (instead of computing all the pairwise similarities in advance), that set of 10 vectors will be missing documents that have a higher cosine similarity than that of the K&amp;#x27;th vector retrieved.&lt;p&gt;All of this means you&amp;#x27;ll need to retrieve a multiple of K vectors, figure out some way to re-rank them to exclude the irrelevant ones, and have your own ground truth to measure the index&amp;#x27;s precision and recall.</text></item></parent_chain><comment><author>brigadier132</author><text>The vectors are literally constructed so that cosine similarity &lt;i&gt;is&lt;/i&gt; semantic similarity.&lt;p&gt;&amp;gt; second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that&amp;#x27;s not true either&lt;p&gt;Its not unstated, its called ANN for a reason</text></comment>
<story><title>LLMs, RAG, and the missing storage layer for AI</title><url>https://blog.lancedb.com/llms-rag-the-missing-storage-layer-for-ai-28ded35fa984</url></story><parent_chain><item><author>panarky</author><text>The first unstated assumption is that similar vectors are relevant documents, and for many use cases that&amp;#x27;s just not true. Cosine similarity != relevance. So if your pipeline pulls 2 or 4 or 12 document chunks into the LLM&amp;#x27;s context, and half or more of them aren&amp;#x27;t relevant, does this make the LLM&amp;#x27;s response more or less relevant?&lt;p&gt;The second unstated assumption is that the vector index can accurately identify the top K vectors by cosine similarity, and that&amp;#x27;s not true either. If you retrieve the top K vectors according to the vector index (instead of computing all the pairwise similarities in advance), that set of 10 vectors will be missing documents that have a higher cosine similarity than that of the K&amp;#x27;th vector retrieved.&lt;p&gt;All of this means you&amp;#x27;ll need to retrieve a multiple of K vectors, figure out some way to re-rank them to exclude the irrelevant ones, and have your own ground truth to measure the index&amp;#x27;s precision and recall.</text></item></parent_chain><comment><author>NhanH</author><text>Could you please explain a bit on your 2nd paragraph. I couldn’t quite understand either the problem statement nor the reasoning itself.</text></comment>