Due to the excellent performance and the long-term decay property, RoPE is widely adopted in the latest LLMs, _e.g._, PaLM [56] and LLMA [57]. Based on RoPE, xPos [277] further improves the translation invariance and length extrapolation of Transformer. At each dimension of the rotation angle vector, xPos adds a special exponential decay that is smaller when the basis is larger. It can alleviate the unstable phenomenon during training as the distance increases.