<story><title>Hardware Acceleration of LLMs: A comprehensive survey and comparison</title><url>https://arxiv.org/abs/2409.03384</url></story><parent_chain><item><author>bilsbie</author><text>Thanks for the background. Whatever happened to memristors and the promise of memory living alongside cpu?</text></item><item><author>refibrillator</author><text>This paper is light on background so I’ll offer some additional context:&lt;p&gt;As early as the 90s it was observed that CPU speed (FLOPs) was improving faster than memory bandwidth. In 1995 William Wulf and Sally Mckee predicted this divergence would lead to a “memory wall”, where most computations would be bottlenecked by data access rather than arithmetic operations.&lt;p&gt;Over the past 20 years peak server hardware FLOPS has been scaling at 3x every 2 years, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively.&lt;p&gt;Thus for training and inference of LLMs, the performance bottleneck is increasingly shifting toward memory bandwidth. Particularly for autoregressive Transformer decoder models, it can be the &lt;i&gt;dominant&lt;/i&gt; bottleneck.&lt;p&gt;This is driving the need for new tech like Compute-in-memory (CIM), also known as processing-in-memory (PIM). Hardware in which operations are performed directly on the data in memory, rather than transferring data to CPU registers first. Thereby improving latency and power consumption, and possibly sidestepping the great “memory wall”.&lt;p&gt;Notably to compare ASIC and FPGA hardware across varying semiconductor process sizes, the paper uses a fitted polynomial to extrapolate to a common denominator of 16nm:&lt;p&gt;&lt;i&gt;&amp;gt; Based on the article by Aaron Stillmaker and B.Baas titled ”Scaling equations for the accurate prediction of CMOS device performance from 180 nm to 7nm,” we extrapolated the performance and the energy efficiency on a 16nm technology to make a fair comparison&lt;/i&gt;&lt;p&gt;But extrapolation for CIM&amp;#x2F;PIM is not done because they claim:&lt;p&gt;&lt;i&gt;&amp;gt; As the in-memory accelerators the performance is not based only on the process technology, the extrapolation is performed only on the FPGA and ASIC accelerators where the process technology affects significantly the performance of the systems.&lt;/i&gt;&lt;p&gt;Which strikes me as an odd claim at face value, but perhaps others here could offer further insight on that decision.&lt;p&gt;Links below for further reading.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2403.14123&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2403.14123&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;In-memory_processing&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;In-memory_processing&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;vcl.ece.ucdavis.edu&amp;#x2F;pubs&amp;#x2F;2017.02.VLSIintegration.TechScale&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;vcl.ece.ucdavis.edu&amp;#x2F;pubs&amp;#x2F;2017.02.VLSIintegration.Tech...&lt;/a&gt;</text></item></parent_chain><comment><author>chatmasta</author><text>I&amp;#x27;m a layman on this topic, so I&amp;#x27;m definitely about to say something wrong. But I recall an intriguing idea about a sort of &amp;quot;reversion to analog,&amp;quot; whereby we use the full range of voltage crossing a resistor. Instead of cutting it in half to produce binary (high voltage is 1, low voltage is 0), we could treat the voltage as a scalar weight within a network of resistors.&lt;p&gt;Has anyone else heard of this idea or have any insight on it?</text></comment>
<story><title>Hardware Acceleration of LLMs: A comprehensive survey and comparison</title><url>https://arxiv.org/abs/2409.03384</url></story><parent_chain><item><author>bilsbie</author><text>Thanks for the background. Whatever happened to memristors and the promise of memory living alongside cpu?</text></item><item><author>refibrillator</author><text>This paper is light on background so I’ll offer some additional context:&lt;p&gt;As early as the 90s it was observed that CPU speed (FLOPs) was improving faster than memory bandwidth. In 1995 William Wulf and Sally Mckee predicted this divergence would lead to a “memory wall”, where most computations would be bottlenecked by data access rather than arithmetic operations.&lt;p&gt;Over the past 20 years peak server hardware FLOPS has been scaling at 3x every 2 years, outpacing the growth of DRAM and interconnect bandwidth, which have only scaled at 1.6 and 1.4 times every 2 years, respectively.&lt;p&gt;Thus for training and inference of LLMs, the performance bottleneck is increasingly shifting toward memory bandwidth. Particularly for autoregressive Transformer decoder models, it can be the &lt;i&gt;dominant&lt;/i&gt; bottleneck.&lt;p&gt;This is driving the need for new tech like Compute-in-memory (CIM), also known as processing-in-memory (PIM). Hardware in which operations are performed directly on the data in memory, rather than transferring data to CPU registers first. Thereby improving latency and power consumption, and possibly sidestepping the great “memory wall”.&lt;p&gt;Notably to compare ASIC and FPGA hardware across varying semiconductor process sizes, the paper uses a fitted polynomial to extrapolate to a common denominator of 16nm:&lt;p&gt;&lt;i&gt;&amp;gt; Based on the article by Aaron Stillmaker and B.Baas titled ”Scaling equations for the accurate prediction of CMOS device performance from 180 nm to 7nm,” we extrapolated the performance and the energy efficiency on a 16nm technology to make a fair comparison&lt;/i&gt;&lt;p&gt;But extrapolation for CIM&amp;#x2F;PIM is not done because they claim:&lt;p&gt;&lt;i&gt;&amp;gt; As the in-memory accelerators the performance is not based only on the process technology, the extrapolation is performed only on the FPGA and ASIC accelerators where the process technology affects significantly the performance of the systems.&lt;/i&gt;&lt;p&gt;Which strikes me as an odd claim at face value, but perhaps others here could offer further insight on that decision.&lt;p&gt;Links below for further reading.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2403.14123&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;arxiv.org&amp;#x2F;abs&amp;#x2F;2403.14123&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;In-memory_processing&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.m.wikipedia.org&amp;#x2F;wiki&amp;#x2F;In-memory_processing&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;vcl.ece.ucdavis.edu&amp;#x2F;pubs&amp;#x2F;2017.02.VLSIintegration.TechScale&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;vcl.ece.ucdavis.edu&amp;#x2F;pubs&amp;#x2F;2017.02.VLSIintegration.Tech...&lt;/a&gt;</text></item></parent_chain><comment><author>tonetegeatinst</author><text>I believe asianometry did a YouTube video on memristors....might be worth watching.</text></comment>