distances between keys and queries. Different from the relative positional embedding methods like T5 [82], the penalty scores in ALBi are pre-defined without any trainable parameters. Empirical results in [264] have shown that ALBi has a better extrapolation performance on sequences that are longer than those for training than several popular position embedding methods such as sinusoidal PE [22], RoPE [263], and T5 bias [82]. In addition, it has been shown that ALBi can also improve training stability in BLOOM [78].