Self-attention networks (SANs) (Parikh et al., 2016; Lin et al., 2017) have shown promising empirical results in various natural language processing (NLP) tasks, such as machine translation <cite>(Vaswani et al., 2017)</cite> , natural language inference (Shen et al., 2018a) , and acoustic modeling (Sperber et al., 2018) .