<story><title>The Era of 1-bit LLMs: ternary parameters for cost-effective computing</title><url>https://arxiv.org/abs/2402.17764</url></story><parent_chain><item><author>paul_mk1</author><text>Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there).&lt;p&gt;Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work).&lt;p&gt;Helpful hint to authors: you can get down to 0.68 bits &amp;#x2F; weight using a similar technique, good chance this will work for LLMs too.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1606.01981&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1606.01981&lt;/a&gt;&lt;p&gt;This was a passion project of mine in my last few months at IBM research :).&lt;p&gt;I am convinced there is a deep connection to understanding why backprop is unreasonably effective, and the result that you can train low precision DNNs; for those note familiar, the technique is to compute the loss wrt to the low precision parameters (eg project to ternary) but apply the gradient to high precision copy of parameters (known as the straight through estimator). This is a biased estimator and there is no theoretical underpinning for why this should work, but in practice it works well.&lt;p&gt;My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.</text></item><item><author>cs702</author><text>There are two findings I find &lt;i&gt;shocking&lt;/i&gt; in this work:&lt;p&gt;* In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1).&lt;p&gt;* In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise &lt;i&gt;additions&lt;/i&gt; (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the paper for exact details.&lt;p&gt;On existing hardware, the gains in compute and memory efficiency are significant, without performance degradation (as tested by the authors).&lt;p&gt;If the proposed methods are implemented in hardware, we will see &lt;i&gt;even greater gains&lt;/i&gt; in compute and memory efficiency.&lt;p&gt;Wow.</text></item></parent_chain><comment><author>cs702</author><text>Thank you. Others on this thread have addressed the citation-trail issues you raise. I just want to tell you how helpful I find your comment about &lt;i&gt;why&lt;/i&gt; ternary weights ought to work at all without degrading performance:&lt;p&gt;&amp;gt; My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about &lt;i&gt;who connects to who (ie a graph)&lt;/i&gt;, and not about the individual weight values anymore.&lt;p&gt;Your guess sounds and feels &lt;i&gt;right&lt;/i&gt; to me, even if currently there&amp;#x27;s no way to express it formally, with the rigor it deserves.&lt;p&gt;Thank you again for your comment!</text></comment>
<story><title>The Era of 1-bit LLMs: ternary parameters for cost-effective computing</title><url>https://arxiv.org/abs/2402.17764</url></story><parent_chain><item><author>paul_mk1</author><text>Fun to see ternary weights making a comeback. This was hot back in 2016 with BinaryConnect and TrueNorth chip from IBM research (disclosure, I was one of the lead chip architects there).&lt;p&gt;Authors seemed to have missed the history. They should at least cite Binary Connect or Straight Through Estimators (not my work).&lt;p&gt;Helpful hint to authors: you can get down to 0.68 bits &amp;#x2F; weight using a similar technique, good chance this will work for LLMs too.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1606.01981&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;1606.01981&lt;/a&gt;&lt;p&gt;This was a passion project of mine in my last few months at IBM research :).&lt;p&gt;I am convinced there is a deep connection to understanding why backprop is unreasonably effective, and the result that you can train low precision DNNs; for those note familiar, the technique is to compute the loss wrt to the low precision parameters (eg project to ternary) but apply the gradient to high precision copy of parameters (known as the straight through estimator). This is a biased estimator and there is no theoretical underpinning for why this should work, but in practice it works well.&lt;p&gt;My best guess is that it is encouraging the network to choose good underlying subnetworks to solve the problem, similar to Lottery Ticket Hypothesis. With ternary weights it is just about who connects to who (ie a graph), and not about the individual weight values anymore.</text></item><item><author>cs702</author><text>There are two findings I find &lt;i&gt;shocking&lt;/i&gt; in this work:&lt;p&gt;* In existing LLMs, we can replace all parameter floating-point values representing real numbers with ternary values representing (-1, 0, 1).&lt;p&gt;* In matrix multiplications (e.g., weights by vectors), we can replace elementwise products in each dot product (a₁b₁ + a₂b₂ ...) with elementwise &lt;i&gt;additions&lt;/i&gt; (a₁+b₁ + a₂+b₂ ...), in which signs depend on each value. See the paper for exact details.&lt;p&gt;On existing hardware, the gains in compute and memory efficiency are significant, without performance degradation (as tested by the authors).&lt;p&gt;If the proposed methods are implemented in hardware, we will see &lt;i&gt;even greater gains&lt;/i&gt; in compute and memory efficiency.&lt;p&gt;Wow.</text></item></parent_chain><comment><author>mjcohen</author><text>IIRC, Hamming&amp;#x27;s book &amp;quot;Digital Filters&amp;quot; (1989) has a section on FFTs with only the sign of the coefficient being used. It performed surprisingly well.</text></comment>