Next, we show that this new formulation allows us to explore new family of attention while at the same time offering a framework to categorize previous attention variants (Vaswani et al., 2017; Shaw et al., 2018; Huang et al., 2018b; <cite>Dai et al., 2019</cite>; Child et al., 2019; Wang et al., 2018; Tsai et al., 2019a) .