<story><title>Chiplet ASIC supercomputers for LLMs like GPT-4</title><url>https://arxiv.org/abs/2307.02666</url></story><parent_chain><item><author>alain94040</author><text>The key point:&lt;p&gt;&lt;i&gt;A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs&lt;/i&gt;&lt;p&gt;...&lt;p&gt;&lt;i&gt;On-chip memories such as SRAM have better read latency and read&amp;#x2F;write energy than external memories such as DDR or HBM but require more silicon per bit. We show this design choice wins in the competition of TCO per performance for serving large generative language models but requires careful consideration with respect to the chiplet die size, chiplet memory capacity and total number of chiplets to balance the fabrication cost and model performance (Sec.3.2.2) We observe that the inter-chiplet communication issues can be effectively mitigated through proper software-hardware co- design leveraging mapping strategies such as tensor and pipeline model parallelism&lt;/i&gt;</text></item></parent_chain><comment><author>senttoschool</author><text>SRAM has stopped scaling based on TSMC&amp;#x27;s upcoming N3E specs and their planned N2 node specs. So if models are tens of GB large, then I don&amp;#x27;t see how their proposed chips can be done in an economical way.&lt;p&gt;Also, a GPU is already an ASIC but with a fancy name.</text></comment>
<story><title>Chiplet ASIC supercomputers for LLMs like GPT-4</title><url>https://arxiv.org/abs/2307.02666</url></story><parent_chain><item><author>alain94040</author><text>The key point:&lt;p&gt;&lt;i&gt;A key architectural feature to achieve this is the ability to fit all model parameters inside the on-chip SRAMs of the chiplets to eliminate bandwidth limitations. Doing so is non-trivial as the amount of memory required is very large and growing for modern LLMs&lt;/i&gt;&lt;p&gt;...&lt;p&gt;&lt;i&gt;On-chip memories such as SRAM have better read latency and read&amp;#x2F;write energy than external memories such as DDR or HBM but require more silicon per bit. We show this design choice wins in the competition of TCO per performance for serving large generative language models but requires careful consideration with respect to the chiplet die size, chiplet memory capacity and total number of chiplets to balance the fabrication cost and model performance (Sec.3.2.2) We observe that the inter-chiplet communication issues can be effectively mitigated through proper software-hardware co- design leveraging mapping strategies such as tensor and pipeline model parallelism&lt;/i&gt;</text></item></parent_chain><comment><author>JonChesterfield</author><text>Graphcore got up to a gigabyte or so of on chip memory with the same plan of keeping the model in that memory. Does work really well if the data fits.&lt;p&gt;Recent x64 chips are at about that amount of L3 cache which might be pretty similar. I&amp;#x27;ve lost track of GPU hardware specs.&lt;p&gt;That proper hardware software co-design to mitigate communication? Viciously difficult bordering on imaginary.</text></comment>