Next, we show that this new formulation allows us to explore new family of attention while at the same time offering a framework to categorize previous attention variants <cite>(Vaswani et al., 2017</cite>; Shaw et al., 2018; Huang et al., 2018b; Dai et al., 2019; Child et al., 2019; Wang et al., 2018; Tsai et al., 2019a) .