We present Nemotron-4 15B, a decoder-only transformer-based large language model. It is trained on 8 trillion tokens spanning English, 53 additional natural languages as well as 43 programming languages. Nemotron-4 15B exhibits the strongest multilingual performance of any general purpose language model at its scale - even outperforming models specialized for the multilingual domain. Nemotron-4 demonstrates that pre-training sets for large language models can continue to be scaled up even further in order to improve the abilities of models.