<story><title>Nvidia releases new AI chip with 480GB CPU RAM, 96GB GPU RAM</title><url>https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/</url></story><parent_chain><item><author>samwillis</author><text>This may be a naive question, in &amp;quot;crypto&amp;quot; we saw a shift from GPUs to ASICs as it was more efficient to design and run chips specifically for hashing. Will we see the same in ML, will there be a shift to ASICs for training and inference of models?&lt;p&gt;Apple already have the &amp;quot;neural&amp;quot; cores, is that more or less what they are?&lt;p&gt;Could there be a theoretical LLM chip for inference that is significantly cheaper to run?</text></item></parent_chain><comment><author>anonylizard</author><text>Inference is mostly just matrix multiplications, so there&amp;#x27;s plenty of competitors.&lt;p&gt;Problem is, inference costs do not dominate training costs. Models have a very limited lifespan, they are constantly retrained or obsoleted by new generations, so training is always going on.&lt;p&gt;Training is not just matrix multiplications, given hundreds of experiments in model architecture, its not even obvious what operations will dominate future training. So a more general purpose GPU is just a way safer bet.&lt;p&gt;Also, LLM talent is in extreme short supply, and you don&amp;#x27;t want to piss them off by telling them they have to spend their time debugging some crappy FPGA because you wanted to save some hardware bucks.</text></comment>
<story><title>Nvidia releases new AI chip with 480GB CPU RAM, 96GB GPU RAM</title><url>https://www.nvidia.com/en-us/data-center/grace-hopper-superchip/</url></story><parent_chain><item><author>samwillis</author><text>This may be a naive question, in &amp;quot;crypto&amp;quot; we saw a shift from GPUs to ASICs as it was more efficient to design and run chips specifically for hashing. Will we see the same in ML, will there be a shift to ASICs for training and inference of models?&lt;p&gt;Apple already have the &amp;quot;neural&amp;quot; cores, is that more or less what they are?&lt;p&gt;Could there be a theoretical LLM chip for inference that is significantly cheaper to run?</text></item></parent_chain><comment><author>huijzer</author><text>According to Ilya Sutskever in a podcast that I heard, GPUs are already pretty close to ASIC performance for AI workloads. Nvidia can highly optimize the full stack due to their economies of scale.</text></comment>