**RoPE Embeddings**(Su et al., 2021). Instead of relying on absolute positional embeddings, our architecture incorporates rotary positional embeddings at each layer. Furthermore, to minimize the overall model size, embeddings are shared between inputs and outputs.