The OpenLLaMA 7Bv2 model, trained on a composite dataset comprising Falcon refined-web and starcoder datasets, augmented by contributions from Wikipedia, arXiv, books, and Stack Exchange data curated by RedPajama. The training utilizes a max learning rate of 3e-4 and min learning rate of 3e-5 with a batch size of 4 million tokens. For the learning rate scheduler, this configuration closely aligns with the setup employed in Llama2.