The English corpus consists of curated documents from a variety of sources and domains including web documents, news articles, scientific papers, books, etc and the distribution used in our pre-training set is highlighted in Figure 2. The code and multilingual data consists of a diverse set of natural and programming languages. We find that appropriately sampling tokens from these languages is key to strong accuracies in these domains. We share the distributions used for both code and multilingual tokens in our pre-training dataset in Figure 3 and Figure 4 respectively.