<story><title>What happens when you vectorize wide PyTorch expressions?</title><url>https://probablymarcus.com/blocks/2023/10/19/vectorizing-wide-pytorch-expressions.html</url></story><parent_chain></parent_chain><comment><author>gregjm</author><text>&amp;gt; My so-called CPU “active” time is actually an inferred value; CUDA spins the CPU 100% constantly, even when the CPU is just waiting for the GPU&lt;p&gt;The CUDA Runtime and Driver APIs allow you to use“blocking synchronization” where the CPU will go to sleep while waiting for synchronization with the device. However, it seems that PyTorch doesn’t expose this functionality in any of its Python APIs:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;pytorch&amp;#x2F;pytorch&amp;#x2F;issues&amp;#x2F;28224&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;pytorch&amp;#x2F;pytorch&amp;#x2F;issues&amp;#x2F;28224&lt;/a&gt;&lt;p&gt;What happens when you try using ctypes to call into libcudart.so to set the device flags as described in the above issue? You’ll have to call torch.cuda.init() for it to work, and unfortunately it won’t work if PyTorch is launching kernels from other threads.</text></comment>
<story><title>What happens when you vectorize wide PyTorch expressions?</title><url>https://probablymarcus.com/blocks/2023/10/19/vectorizing-wide-pytorch-expressions.html</url></story><parent_chain></parent_chain><comment><author>pixelpoet</author><text>I really hope those pow(x, 2) calls are getting turned into x * x, else it&amp;#x27;s a performance catastrophe &amp;#x2F; extreme beginner mistake even with vectorisation.&lt;p&gt;Also, this kind of ultra wide buffering consumes a ton of memory bandwidth for each operation, instead of keeping a small portion in cache&amp;#x2F;registers. FLOPs are scaling sort of infinitely, whereas memory speed is flat, so this is increasingly a losing game; just because it&amp;#x27;s faster than glacial Python doesn&amp;#x27;t mean it&amp;#x27;s fast compared to a language which actually concerns itself with performance or a more cache aware approach.&lt;p&gt;For an extreme example of how you can even sometimes beat ultra optimised GPU ML libraries in this way, check out &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;NVlabs&amp;#x2F;tiny-cuda-nn&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;NVlabs&amp;#x2F;tiny-cuda-nn&lt;/a&gt;</text></comment>