Nemotron-4 uses a standard decoder-only Transformer architecture (Vaswani et al., 2017), with causal attention masks. Exact hyper-parameters affecting size are shown in Table 1. Nemotron-4 has 3.2 billion embedding parameters and 12.5 billion non-embedding parameters. We use Rotary Position Embeddings (RoPE) (Su et al., 2021), SentencePiece tokenizer (Kudo and Richardson, 2018), squared ReLU activations in the MLP layers, no bias terms, dropout rate of zero, and untied input-output embeddings.