<story><title>Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer</title><url>https://www.hpcwire.com/2021/05/27/nersc-debuts-perlmutter-worlds-fastest-ai-supercomputer/</url></story><parent_chain></parent_chain><comment><author>boulos</author><text>Disclosure: I used to work on Google Cloud.&lt;p&gt;Perlmutter seems like an awesome system. But, I think the “ai exaflops” is a “X GPUS times the NVIDIA peak rate”. The new sparsity features on A100 are promising, but haven’t been demonstrated to be nearly as awesome in practice (yet).&lt;p&gt;It also all comes down to workloads: large-scale distributed training is a funny workload! It’s &lt;i&gt;not&lt;/i&gt; like LINPACK. If you make your model compute intensive enough, then the networking need mostly becomes &lt;i&gt;bandwidth&lt;/i&gt; (for which multi-hundred Gbps worth of NICs is handy) but even without it there are lots of ways to max out your compute.&lt;p&gt;Similarly, &lt;i&gt;storage&lt;/i&gt; is a serious need for say giant video corpora, but not for things like text! GPT-2 had like a 40 GiB corpus.&lt;p&gt;For those asking about largest-scale cloud &lt;i&gt;GPU&lt;/i&gt; runs, there are basically three examples (chronological)&lt;p&gt;- the work OpenAI did on Five (thousands of V100s on GCP)&lt;p&gt;- the IceCube science work [1] on many clouds (51000 GPUs at peak!)&lt;p&gt;- OpenAI’s 10000-V100 cluster they used for GPT-3&lt;p&gt;The &lt;i&gt;A100&lt;/i&gt;s used here are recently released and another step-change in perf per part (and memory). All major providers now offer them, though with different density and networking configurations (GCP went with 16 in a single box, most folks went with 8, some have lots of networking, etc.).&lt;p&gt;What everyone should be asking is: what awesome stuff is NERSC &amp;#x2F; LBL going to do with Perlmutter? You can’t just rent one for a few hours on GCP or any other provider :). (But, fwiw, most usage will be small slices: this is the sad fate of giant supercomputers!)&lt;p&gt;[1] &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;insidehpc.com&amp;#x2F;2019&amp;#x2F;11&amp;#x2F;sdsc-conducts-50000-gpu-cloudburst-experiment-with-wisconsin-icecube-particle-astrophysics-center&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;insidehpc.com&amp;#x2F;2019&amp;#x2F;11&amp;#x2F;sdsc-conducts-50000-gpu-cloudb...&lt;/a&gt;</text></comment>
<story><title>Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer</title><url>https://www.hpcwire.com/2021/05/27/nersc-debuts-perlmutter-worlds-fastest-ai-supercomputer/</url></story><parent_chain></parent_chain><comment><author>mooneater</author><text>&amp;gt; Python programmers will be able to use RAPIDS, Nvidia’s open software suite for GPU-enabled data science&lt;p&gt;Ah not to be confused with OpenAI Rapid described at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;openai.com&amp;#x2F;blog&amp;#x2F;openai-five&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;openai.com&amp;#x2F;blog&amp;#x2F;openai-five&amp;#x2F;&lt;/a&gt; ^_^</text></comment>