<story><title>The One Billion Row Challenge in CUDA</title><url>https://tspeterkim.github.io/posts/cuda-1brc</url></story><parent_chain><item><author>_zoltan_</author><text>this is true for small datasets.&lt;p&gt;on large datasets, once loaded into GPU memory, cross GPU shuffling with NVLink is going to be much faster than CPU to RAM.&lt;p&gt;on the H100 boxes with 8x400Gbps, IO with GDS is also pretty fast.&lt;p&gt;for truly IObound tasks I think a lot of GPUs beats almost anything :-)</text></item><item><author>pama</author><text>There are some good ideas for this type of problem here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;dannyvankooten&amp;#x2F;1brc&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;dannyvankooten&amp;#x2F;1brc&lt;/a&gt;&lt;p&gt;After you deal with parsing and hashes, basically you are IO limited, so mmap helps. The C code takes less than 1.4s without any CUDA access. Because there is no compute to speak of, other than parsing and a hashmap, a reasonable guess is that even for the optimal CUDA implementation, the starting of kernels and transfer of data to the GPU would likely add a noticeable bottleneck and make the optimal CUDA code slower than this pure C code.</text></item></parent_chain><comment><author>pama</author><text>Yes, GDS will accelerate the IO to the GPU. I’d love to see the above C code compared to hyperoptimized GPU code on the right hardware, but I don’t want to accidentally nerd snipe myself :-)  The unfortunate part of this particular benchmark is that once you have the data in the right place in your hardware there is very little compute left. The GPU code would probably have constant performance with an additional couple thousand operations on each row whereas CPU would slow down.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;docs.nvidia.com&amp;#x2F;gpudirect-storage&amp;#x2F;overview-guide&amp;#x2F;index.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;docs.nvidia.com&amp;#x2F;gpudirect-storage&amp;#x2F;overview-guide&amp;#x2F;ind...&lt;/a&gt;</text></comment>
<story><title>The One Billion Row Challenge in CUDA</title><url>https://tspeterkim.github.io/posts/cuda-1brc</url></story><parent_chain><item><author>_zoltan_</author><text>this is true for small datasets.&lt;p&gt;on large datasets, once loaded into GPU memory, cross GPU shuffling with NVLink is going to be much faster than CPU to RAM.&lt;p&gt;on the H100 boxes with 8x400Gbps, IO with GDS is also pretty fast.&lt;p&gt;for truly IObound tasks I think a lot of GPUs beats almost anything :-)</text></item><item><author>pama</author><text>There are some good ideas for this type of problem here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;dannyvankooten&amp;#x2F;1brc&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;dannyvankooten&amp;#x2F;1brc&lt;/a&gt;&lt;p&gt;After you deal with parsing and hashes, basically you are IO limited, so mmap helps. The C code takes less than 1.4s without any CUDA access. Because there is no compute to speak of, other than parsing and a hashmap, a reasonable guess is that even for the optimal CUDA implementation, the starting of kernels and transfer of data to the GPU would likely add a noticeable bottleneck and make the optimal CUDA code slower than this pure C code.</text></item></parent_chain><comment><author>candido_heavyai</author><text>I am testing a gh200 and the speed you can access the system memory is amazing.. Assuming you have already encoded the station into a smallint and  the size of the dataset would be around 6gb that on such system takes just  20 ms to be transfered (I am sure about that because I&amp;#x27;m observing transfer a 9.5gb that took about 33ms right now).</text></comment>