Pre-training.Nemotron-4 was trained using 384 DGX H100 nodes; each node contains 8 H100 80GB SXM5 GPUs based on the NVIDIA Hopper architecture (NVIDIA, 2022). Each H100 GPU has a peak throughput of 989 teraFLOP/s when doing 16-bit floating point (bfloat16) arithmetic without sparsity. Within each node, GPUs are connected by NVLink and NVSwitch (nvl); the GPU-to-GPU bandwidth is 900 GB/s (450 GB/s in each direction). Each node has 8 NVIDIA Mellanox 400 Gbps HDR InfiniBand Host Channel Adapters (HCAs) for inter-node communication.