We trained for approximately 131 hours or 5.5 days on a 128-core TPU v3. The approximate deduplicated dataset is 3.9% smaller than the original dataset and trains in 63 hours/epoch, saving us around 5 hours of compute time for the two epochs. The XL-Originalmodel was trained in North America where the XL-ExactSubstr and XL-NearDup were trained in Taiwan. We used data from Patterson et al. (2021) to estimate amount of energy used in training these models by computing the amount of \(MWh\)/hour/core and multiplying by our usage (see Table 6 for how we computed these values). For simplicity, we use estimates from Taiwaninese datacenters as an estimate. We estimate training 2 epochs of XL-Original and XL-ExactSubstr uses \(5.86MWh\). XL-NearDup is trained for fewer steps and we estimate uses \(5.63MWh\). Training each base model was approximately 3 days on a 64-core TPU v3 pod slice which uses an estimated \(1.61MWh\).