<story><title>LLM in a Flash: Efficient LLM Inference with Limited Memory</title><url>https://huggingface.co/papers/2312.11514</url></story><parent_chain></parent_chain><comment><author>PoignardAzur</author><text>It took me a while to understand that paper, because it builds on the techniques of the &lt;i&gt;Deja Vu&lt;/i&gt; paper for leveraging sparsity which are already pretty complex:&lt;p&gt;- First, the &lt;i&gt;Deja Vu&lt;/i&gt; paper observes that models with low weight sparsity have what they call high &amp;quot;contextual sparsity&amp;quot;. Basically, the matrix multiplications will produce vectors with lots of zeros in them, but where the zeros are depends on the input.&lt;p&gt;- The paper notes that you can use that sparsity to skip loading some rows of your matrices.&lt;p&gt;- However, to get good performance benefits, you can to &lt;i&gt;predict&lt;/i&gt; in advance which rows you&amp;#x27;re going to skip. You can do that with a low-rank matrix.&lt;p&gt;The Apple paper then suggests these findings can not just improve your performance loading from RAM, but even allow you to load from flash memory without sacrificing bandwidth:&lt;p&gt;- The paper notes that attention matrices are rather light; the FFNs are the ones you want to load sparsely.&lt;p&gt;- The paper notes that you can get much better sparsity by predicting the output of the ReLU layer rather than the input of the FFN. Basically saying &amp;quot;if you can predict that after matmul this slot of the vector will have a negative value before ReLU, you can skip loading the matrix column and output zero&amp;quot;.&lt;p&gt;- The paper suggests that you don&amp;#x27;t need to load most rows of the FFN &lt;i&gt;at all&lt;/i&gt;; you can just keep a cache of recently used FFN rows for each FFN, and update it from flash memory on-demand.&lt;p&gt;There&amp;#x27;s a bunch more about chunk loading and the correlation between projection layers, but the above is where I think the main insights are.&lt;p&gt;(FFN = Feed Forward Network; in the context of transformers they&amp;#x27;re the biggest blocks.)</text></comment>
<story><title>LLM in a Flash: Efficient LLM Inference with Limited Memory</title><url>https://huggingface.co/papers/2312.11514</url></story><parent_chain></parent_chain><comment><author>MBCook</author><text>I wonder how much of the model you can avoid loading before you start to see a real performance difference.&lt;p&gt;Let’s say you want to maintain 90% of everything-in-RAM performance.&lt;p&gt;Can you get away with only using half the memory? Do you need 90% of the memory? Maybe 95%?&lt;p&gt;Basically how fast do you lose performance compared to the maximum by cutting RAM. The charts are comparing their algorithm vs a basic one for the less RAM case, which is a different (but good!) question.&lt;p&gt;If you can get good performance by NOT loading an entire eight gig model into memory of a cell phone that’s obviously a very useful thing.</text></comment>