Our model's architecture is based on the transformer decoder (Vaswani et al., 2017). The key parameters that define our architecture are shown in Table 1, with the models being trained on a substantial context length of 4096 tokens. Beyond the foundational elements, our approach integrates several improvements compared to the original transformer.