While creating our tokenizer and model training datasets, we ensure that the mixture proportions of data domains (Wikipedia, books, web text) follow the same distribution to avoid a domain shift between tokenizers training and model training. We created _two datasets_ with 70B words where one of the datasets is monolingual, containing English documents, and the second is a multilingual dataset comprised of English, German, French, Italian, and Spanish documents. Our datasets are filtered and deduplicated and consist of web-crawled data (80%) and curated data (20%), comparable to related datasets used to train LLMs. In the multilingual dataset, the amount of web-crawled data is equally distributed across languages in terms of number of words. Further details about our data pipeline and the data composition are described in Appendix A.