<story><title>Llama.cpp can do 40 tok/s on M2 Max, 0% CPU usage, using all 38 GPU cores</title><url>https://twitter.com/natfriedman/status/1665402680376987648</url></story><parent_chain><item><author>samwillis</author><text>In only a few years we are going to have close to this level of hardware on all laptops&amp;#x2F;desktops and maybe even mobiles. Manufactures such as Apple will be driven by consumer desire for local LLMs, that are privet and fine tuned with personal data, to move in that direction.&lt;p&gt;Apple are particularly well placed to do this with the unified memory architecture of the M series processors.&lt;p&gt;Local LLMs are the future, not in the cloud, I really wouldn&amp;#x27;t be supprised if Apple drop something along these lines on Monday. I will be looking out for a &lt;i&gt;&amp;quot;oh, just one more thing&amp;quot;&lt;/i&gt; after the headset demos.</text></item></parent_chain><comment><author>rcarmo</author><text>Their hardware development cycles are anything between 2-7 years, and even if the M3 was literally twice the performance of the M2, they would be unlikely to boost baseline RAM and storage on _any_ device to do that kind of thing this year.&lt;p&gt;Over time, sure. Maybe even on the top-tier Macs as a demo. But I wouldn’t expect anything better than LLaMa.&lt;p&gt;I would, however, expect them to come to their senses regarding how inadequate Siri is.</text></comment>
<story><title>Llama.cpp can do 40 tok/s on M2 Max, 0% CPU usage, using all 38 GPU cores</title><url>https://twitter.com/natfriedman/status/1665402680376987648</url></story><parent_chain><item><author>samwillis</author><text>In only a few years we are going to have close to this level of hardware on all laptops&amp;#x2F;desktops and maybe even mobiles. Manufactures such as Apple will be driven by consumer desire for local LLMs, that are privet and fine tuned with personal data, to move in that direction.&lt;p&gt;Apple are particularly well placed to do this with the unified memory architecture of the M series processors.&lt;p&gt;Local LLMs are the future, not in the cloud, I really wouldn&amp;#x27;t be supprised if Apple drop something along these lines on Monday. I will be looking out for a &lt;i&gt;&amp;quot;oh, just one more thing&amp;quot;&lt;/i&gt; after the headset demos.</text></item></parent_chain><comment><author>smoldesu</author><text>Apple is kinda hamstrung with the M1. Their base model is 8gb, only ~5 of which is addressable by the system &lt;i&gt;if&lt;/i&gt; you have nothing else open. They need to bump up the minimum spec if they want local LLM support to feel seamless, especially on iPhone where the memory constraints are even more dire.&lt;p&gt;Honestly, Apple doesn&amp;#x27;t strike me as any better or worse off than their competitors. Maybe AMD, given how late to the party they are. Intel has AVX fallback acceleration through Haswell though, and libraries like OpenVINO offer extra acceleration paths for modern Intel CPUs. ARM has ARMnn for any multicore ARMv8 system, and of course Nvidia has CUDA support for multiple arches and OS. Projects like the ONNX Runtime are unifying all of these inferencing interfaces, giving developers multiple deployment options without resorting to walled garden implementations. WebGPU may obsolete them altogether.&lt;p&gt;Local LLMs are awesome, but Apple has some work to do if they want to lead the pack.</text></comment>