Furthermore, our proposed formulation highlights naturally the main components of Transformer's attention, enabling a better understanding of this mechanism: recent variants of Transformers (Shaw et al., 2018; Huang et al., 2018b; <cite>Dai et al., 2019</cite>; Child et al., 2019; Wang et al., 2018; Tsai et al., 2019a) can be expressed through these individual components.