\(\bullet\)_Rotary Position Embedding._ Rotary position embedding (RoPE) [263] sets specific rotatory matrices based on the absolute position of each key or query. The scores between keys and queries can be computed with relative position information (Table VI). RoPE combines each consecutive pair of elements in query and key vectors as _a dimension_, so there are \(d/2\) dimensions for an original \(d\)-length embedding. For each dimension \(i\in\{1,\ldots,d/2\}\), the pair of involved elements will rotate based on the rotation angle \(t\cdot\theta_{i}\), where \(t\) denotes the position index and \(\theta_{i}\) is the basis in the dimension.