The new formulation defines a larger space for composing attention by manipulating its individual components, and at the same time it is able to categorize different variants of attention in prior work (Shaw et al., 2018; Huang et al., 2018b; <cite>Dai et al., 2019</cite>; Child et al., 2019; Wang et al., 2018; Tsai et al., 2019a) .