In order to verify that our codebase could be used on both NVIDIA and AMD GPUs without any loss in performance, we trained models on two different clusters:* **LUMI:** Provided by the LUMI supercomputer,4 we used up to 256 nodes on this cluster, where each node consists of 4x AMD MI250X GPUs with 128GB of memory5 and 800Gbps of interconnect. Footnote 4: [https://www.lumi-supercomputer.eu](https://www.lumi-supercomputer.eu)
* **MosaicML:** Provided by MosaicML.6 (Datalysts), we used 27 nodes on this cluster, where each node consists of 8x NVIDIA A100 GPUs with 40GB of memory and 800Gbps interconnect. Footnote 5: The MI250X is a dual-chip module, meaning in practice that each physical device consists of two logical devices, so each node has 8 logical GPU devices with 64GB of memory each.