Here, \(\sigma\) and \(z_{\text{s}-1}\in\mathbb{R}^{d_{key}}\) are a nonlinear activation function and a normalization term, respectively. As the choice of the non-linearity and the norm method is crucial for training stability, following Katharopoulos et al. (2020) we record a sum over all keys as the normalization term \(z_{\text{s}-1}\) and use element-wise ELU + 1 as the activation function (Clevert et al., 2015).