<story><title>Nvidia’s H100: Funny L2, and Tons of Bandwidth</title><url>https://chipsandcheese.com/2023/07/02/nvidias-h100-funny-l2-and-tons-of-bandwidth/</url></story><parent_chain></parent_chain><comment><author>19h</author><text>I’d be careful with H100 on clouds right now — we ran Onnx &amp;amp; PyTorch payloads an ran into issues that haven’t been addressed yet.&lt;p&gt;For instance, onnx &amp;amp; PyTorch just died randomly and a few thousand dollars later we figured it was directly caused by incorrect assumptions made in the underlying implementations where the developers almost definitely did not have access to an H100 yet.</text></comment>
<story><title>Nvidia’s H100: Funny L2, and Tons of Bandwidth</title><url>https://chipsandcheese.com/2023/07/02/nvidias-h100-funny-l2-and-tons-of-bandwidth/</url></story><parent_chain></parent_chain><comment><author>Roark66</author><text>Personally I think we&amp;#x27;re doing AI wrong. Every time we simplify the architecture we get huge improvement in speed. CPUs have to everything (load&amp;#x2F;store, branching, execution), GPUs do a lot less branching and aren&amp;#x27;t optimised for it. This is why they can be made massively parallel. What if we cut out arbitrary load&amp;#x2F;store too? This would look like A DSP. All memory would be prepared in advance and the massively parallel accelerator would &amp;quot;stream walk&amp;quot; through It without having to load store arbitrary memory locations. The speed improvement could be on par with CPU-&amp;gt;Gpu. Of course this would be for inference mostly. There is a startup that is trying to do just that called tinycorp. I&amp;#x27;m certainly watching it with interest</text></comment>