<story><title>Product Quantization: Compressing high-dimensional vectors</title><url>https://www.pinecone.io/learn/product-quantization/</url></story><parent_chain></parent_chain><comment><author>danieldk</author><text>Nice article that explains product quantization very well, thanks!&lt;p&gt;PQ is really a nice compression technique. I implemented PQ and Optimized PQ [1] a while back in our word embedding package for Rust:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;finalfusion&amp;#x2F;finalfusion-rust&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;finalfusion&amp;#x2F;finalfusion-rust&amp;#x2F;&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;finalfusion&amp;#x2F;reductive&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;finalfusion&amp;#x2F;reductive&amp;#x2F;&lt;/a&gt;&lt;p&gt;Particularly Optimized PQ was effective in reducing vector sizes ~10 times with virtually no reconstruction loss. This made it much easier to ship models (no more 3GB embedding matrix with a neural net that is just a few megabytes large).&lt;p&gt;[1] &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;kaiminghe.com&amp;#x2F;publications&amp;#x2F;pami13opq.pdf&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;kaiminghe.com&amp;#x2F;publications&amp;#x2F;pami13opq.pdf&lt;/a&gt;</text></comment>
<story><title>Product Quantization: Compressing high-dimensional vectors</title><url>https://www.pinecone.io/learn/product-quantization/</url></story><parent_chain></parent_chain><comment><author>q_andrew</author><text>Having a passing knowledge of how ML works, I always wondered how these algorithms can use such dense data structures -- it makes a lot more sense now that I know they can be compressed and still searchable.</text></comment>