<story><title>PyTorch Library for Running LLM on Intel CPU and GPU</title><url>https://github.com/intel-analytics/ipex-llm</url></story><parent_chain><item><author>vegabook</author><text>The company that did 4-cores-forever, has the opportunity  to redeem itself, in its next consumer GPU release, by disrupting the &amp;quot;8-16GB VRAM forever&amp;quot; that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point.&lt;p&gt;Intel definitely seems to be doing all the right things on  software support.</text></item></parent_chain><comment><author>riskable</author><text>No kidding...  Intel is playing catch-up with Nvidia in the AI space and a big reason for that is their offerings aren&amp;#x27;t competitive.  You can get an Intel Arc A770 with 16GB of VRAM (which was released in October, 2022) for about $300 or an Nvidia 4060 Ti with 16GB of VRAM for ~$500 which is &lt;i&gt;twice&lt;/i&gt; as fast for AI workloads in reality (see: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;cdn.mos.cms.futurecdn.net&amp;#x2F;FtXkrY6AD8YypMiHrZuy4K-1200-80.png.webp&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;cdn.mos.cms.futurecdn.net&amp;#x2F;FtXkrY6AD8YypMiHrZuy4K-120...&lt;/a&gt; )&lt;p&gt;This is a huge problem because &lt;i&gt;in theory&lt;/i&gt; the Arc A770 is faster!  It&amp;#x27;s theoretical performance (TFLOPS) is &lt;i&gt;more&lt;/i&gt; than twice as fast as an Nvidia 4060 (see: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;cdn.mos.cms.futurecdn.net&amp;#x2F;Q7WgNxqfgyjCJ5kk8apUQE-1200-80.png&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;cdn.mos.cms.futurecdn.net&amp;#x2F;Q7WgNxqfgyjCJ5kk8apUQE-120...&lt;/a&gt; ).  So why does it perform so poorly?  Because everything AI-related has been developed and optimized to run on Nvidia&amp;#x27;s CUDA.&lt;p&gt;Mostly, this is a mindshare issue.  If Intel offered a workstation GPU (i.e. &lt;i&gt;not&lt;/i&gt; a ridiculously expensive &amp;quot;enterprise&amp;quot; monster) that developers could use that had something like 32GB or 64GB of VRAM it would sell!  They&amp;#x27;d sell zillions of them!  In fact, I&amp;#x27;d wager that they&amp;#x27;d be &lt;i&gt;so&lt;/i&gt; popular it&amp;#x27;d be hard for consumers to even get their hands on one because it would sell out everywhere.&lt;p&gt;It doesn&amp;#x27;t even need to be the fastest card.  It just needs to offer more VRAM than the competition.  Right now, if you want to do things like training or video generation the lack of VRAM is a bigger bottleneck than the speed of the GPU.  How does Intel not see this‽  They have the power to step up and take over a huge section of the market but instead they&amp;#x27;re just copying (poorly) what everyone else is doing.</text></comment>
<story><title>PyTorch Library for Running LLM on Intel CPU and GPU</title><url>https://github.com/intel-analytics/ipex-llm</url></story><parent_chain><item><author>vegabook</author><text>The company that did 4-cores-forever, has the opportunity  to redeem itself, in its next consumer GPU release, by disrupting the &amp;quot;8-16GB VRAM forever&amp;quot; that AMD and Nvidia have been imposing on us for a decade. It would be poetic to see 32-48GB at a non-eye-watering price point.&lt;p&gt;Intel definitely seems to be doing all the right things on  software support.</text></item></parent_chain><comment><author>chessgecko</author><text>Going above 24GB is probably not going to be cheap until gddr7 is out, and even that will only push it to 36gb. The fancier stacked gddr6 stuff is probably pretty expensive and you can’t just add more dies because of signal integrity issues.</text></comment>