We used a combination of 8-way tensor parallelism (Shoeybi et al., 2019) and data parallelism to train the model; we also use a distributed optimizer to shard the optimizer state over the data-parallel replicas. The degree of data parallelism was varied from 96 to 384 as the batch size was ramped up. Table 2 summarizes the 3 stages of batch size ramp, and includes the per-iteration time and model FLOP/s utilization (MFU) (Chowdhery et al., 2022; Korthikanti et al., 2022). MFU quantifies how efficiently the GPUs are utilized in model training. Training was completed in approximately 13 calendar days.