While final accuracy provides a good measure of performance on the pre-training objective, LLMs abilities are typically judged by their performance on evaluation tasks. With the caveat that we use base models, that is our models have not been instruction-tuned, fine-tuned, or adapted to human preferences in any way, we present their evaluation on popular benchmarks in this section. We refer the reader to Sec. 5.4 of the main manuscript and Sec. A.5 of the appendix for a more detailed description of the chosen evaluation tasks.