Furthermore, our proposed formulation highlights naturally the main components of <cite>Transformer's</cite> attention, enabling a better understanding of this mechanism: recent variants of <cite>Transformers</cite> (Shaw et al., 2018; Huang et al., 2018b; Dai et al., 2019; Child et al., 2019; Wang et al., 2018; Tsai et al., 2019a) can be expressed through these individual components.