<story><title>Zen, CUDA, and Tensor Cores, Part I: The Silicon</title><url>https://www.computerenhance.com/p/zen-cuda-and-tensor-cores-part-i</url></story><parent_chain><item><author>raphlinus</author><text>Here&amp;#x27;s my quick take.&lt;p&gt;A top of the line Zen core is a powerful CPU with wide SIMD (AVX-512 is 16 lanes of 32 bit quantities), significant superscalar parallelism (capable of issuing approximately 4 SIMD operations per clock), and a high clock rate (over 5GHz). There isn&amp;#x27;t a lot of confusion about what constitutes a &amp;quot;core,&amp;quot; though multithreading can inflate the &amp;quot;thread&amp;quot; count. See [1] for a detailed analysis of the Zen 5 line.&lt;p&gt;A single Granite Ridge core has peak 32 bit multiply-add performance of about 730 GFLOPS.&lt;p&gt;Nvidia, by contrast, uses the marketing term &amp;quot;core&amp;quot; to refer to a single SIMD lane. Their GPUs are organized as 32 SIMD lanes grouped into each &amp;quot;warp,&amp;quot; and 4 warps grouped into a Streaming Multiprocessor (SM). CPU and GPU architectures can&amp;#x27;t be directly compared, but just going by peak floating point performance, the most comparable granularity to a CPU core is the SM. A warp is in some ways more powerful than a CPU core (generally wider SIMD, larger register file, more local SRAM, better latency hiding) but in other ways less (much less superscalar parallelism, lower clock, around 2.5GHz). A 4090 has 128 SMs, which is a lot and goes a long way to explaining why a GPU has so much throughput. A 1080, by contrast, has 20 SMs - still a goodly number but not mind-meltingly bigger than a high end CPU. See the Nvidia Ada whitepaper [2] for an extremely detailed breakdown of 4090 specs (among other things).&lt;p&gt;A single Nvidia 4090 &amp;quot;core&amp;quot; has peak 32 bit multiply-add performance of about 5 GFLOPS, while an SM has 640 GFLOPS.&lt;p&gt;I don&amp;#x27;t know anybody who counts tensor cores by core count, as the capacity of a &amp;quot;core&amp;quot; varies pretty widely by generation. It&amp;#x27;s almost certainly best just to compare TFLOPS - also a bit of a slippery concept, as that depends on the precision and also whether the application can make use of the sparsity feature.&lt;p&gt;I&amp;#x27;ll also note that not all GPU vendors follow Nvidia&amp;#x27;s lead in counting individual SIMD lanes as &amp;quot;cores.&amp;quot; Apple Silicon, by contrast, uses &amp;quot;core&amp;quot; to refer to a grouping of 128 SIMD lanes, similar to an Nvidia SM. A top of the line M2 Ultra contains 76 such cores, for 9728 SIMD lanes. I found Philip Turner&amp;#x27;s Metal benchmarks [3] useful for understanding the quantitative similarities and differences between Apple, AMD, and Nvidia GPUs.&lt;p&gt;[1]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.numberworld.org&amp;#x2F;blogs&amp;#x2F;2024_8_7_zen5_avx512_teardown&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.numberworld.org&amp;#x2F;blogs&amp;#x2F;2024_8_7_zen5_avx512_teardo...&lt;/a&gt;&lt;p&gt;[2]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;images.nvidia.com&amp;#x2F;aem-dam&amp;#x2F;Solutions&amp;#x2F;Data-Center&amp;#x2F;l4&amp;#x2F;nvidia-ada-gpu-architecture-whitepaper-V2.02.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;images.nvidia.com&amp;#x2F;aem-dam&amp;#x2F;Solutions&amp;#x2F;Data-Center&amp;#x2F;l4&amp;#x2F;n...&lt;/a&gt;&lt;p&gt;[3]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;philipturner&amp;#x2F;metal-benchmarks&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;philipturner&amp;#x2F;metal-benchmarks&lt;/a&gt;</text></item><item><author>fulafel</author><text>The answer to the leading question &amp;quot;What’s the difference between a Zen core, a CUDA core, and a Tensor core?&amp;quot; is not covered in Part 1, so you may want to wait  if this interests you more than chip layouts.</text></item></parent_chain><comment><author>JonChesterfield</author><text>An x64 core roughly corresponding to a SM, or in the amdgpu world a compute unit (CU) seems right. It&amp;#x27;s in the same ballpark for power consumption, represents the component handling an instruction pointer and a local register file and so forth.&lt;p&gt;A really big CPU is a couple of hundred cores, a big GPU is a few hundred SM &amp;#x2F; CUs. Some low power chips are 8 x64 cores and 8 CUs on the same package. All roughly lines up.</text></comment>
<story><title>Zen, CUDA, and Tensor Cores, Part I: The Silicon</title><url>https://www.computerenhance.com/p/zen-cuda-and-tensor-cores-part-i</url></story><parent_chain><item><author>raphlinus</author><text>Here&amp;#x27;s my quick take.&lt;p&gt;A top of the line Zen core is a powerful CPU with wide SIMD (AVX-512 is 16 lanes of 32 bit quantities), significant superscalar parallelism (capable of issuing approximately 4 SIMD operations per clock), and a high clock rate (over 5GHz). There isn&amp;#x27;t a lot of confusion about what constitutes a &amp;quot;core,&amp;quot; though multithreading can inflate the &amp;quot;thread&amp;quot; count. See [1] for a detailed analysis of the Zen 5 line.&lt;p&gt;A single Granite Ridge core has peak 32 bit multiply-add performance of about 730 GFLOPS.&lt;p&gt;Nvidia, by contrast, uses the marketing term &amp;quot;core&amp;quot; to refer to a single SIMD lane. Their GPUs are organized as 32 SIMD lanes grouped into each &amp;quot;warp,&amp;quot; and 4 warps grouped into a Streaming Multiprocessor (SM). CPU and GPU architectures can&amp;#x27;t be directly compared, but just going by peak floating point performance, the most comparable granularity to a CPU core is the SM. A warp is in some ways more powerful than a CPU core (generally wider SIMD, larger register file, more local SRAM, better latency hiding) but in other ways less (much less superscalar parallelism, lower clock, around 2.5GHz). A 4090 has 128 SMs, which is a lot and goes a long way to explaining why a GPU has so much throughput. A 1080, by contrast, has 20 SMs - still a goodly number but not mind-meltingly bigger than a high end CPU. See the Nvidia Ada whitepaper [2] for an extremely detailed breakdown of 4090 specs (among other things).&lt;p&gt;A single Nvidia 4090 &amp;quot;core&amp;quot; has peak 32 bit multiply-add performance of about 5 GFLOPS, while an SM has 640 GFLOPS.&lt;p&gt;I don&amp;#x27;t know anybody who counts tensor cores by core count, as the capacity of a &amp;quot;core&amp;quot; varies pretty widely by generation. It&amp;#x27;s almost certainly best just to compare TFLOPS - also a bit of a slippery concept, as that depends on the precision and also whether the application can make use of the sparsity feature.&lt;p&gt;I&amp;#x27;ll also note that not all GPU vendors follow Nvidia&amp;#x27;s lead in counting individual SIMD lanes as &amp;quot;cores.&amp;quot; Apple Silicon, by contrast, uses &amp;quot;core&amp;quot; to refer to a grouping of 128 SIMD lanes, similar to an Nvidia SM. A top of the line M2 Ultra contains 76 such cores, for 9728 SIMD lanes. I found Philip Turner&amp;#x27;s Metal benchmarks [3] useful for understanding the quantitative similarities and differences between Apple, AMD, and Nvidia GPUs.&lt;p&gt;[1]: &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;www.numberworld.org&amp;#x2F;blogs&amp;#x2F;2024_8_7_zen5_avx512_teardown&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;www.numberworld.org&amp;#x2F;blogs&amp;#x2F;2024_8_7_zen5_avx512_teardo...&lt;/a&gt;&lt;p&gt;[2]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;images.nvidia.com&amp;#x2F;aem-dam&amp;#x2F;Solutions&amp;#x2F;Data-Center&amp;#x2F;l4&amp;#x2F;nvidia-ada-gpu-architecture-whitepaper-V2.02.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;images.nvidia.com&amp;#x2F;aem-dam&amp;#x2F;Solutions&amp;#x2F;Data-Center&amp;#x2F;l4&amp;#x2F;n...&lt;/a&gt;&lt;p&gt;[3]: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;philipturner&amp;#x2F;metal-benchmarks&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;philipturner&amp;#x2F;metal-benchmarks&lt;/a&gt;</text></item><item><author>fulafel</author><text>The answer to the leading question &amp;quot;What’s the difference between a Zen core, a CUDA core, and a Tensor core?&amp;quot; is not covered in Part 1, so you may want to wait  if this interests you more than chip layouts.</text></item></parent_chain><comment><author>Remnant44</author><text>Hi Raph, first of all thank you for all of your contributions and writings - I&amp;#x27;ve learned a ton from reading your blog!&lt;p&gt;A minor quibble amidst your good comparison above ;)&lt;p&gt;For a zen5 core, we have 16-wide SIMD with 4 pipes; 2 are FMA (2 flop), and 2 are FADD @ ~5GHZ. I math that out to 16 * 6 * 5 = 480 GFLOP&amp;#x2F;core... am I missing something?</text></comment>