Transformer (Vaswani et al., 2017 ) is a relative new architecture which outperforms traditional deep learning models such as Recurrent Neural Networks (RNNs) (Sutskever et al., 2014) and Temporal Convolutional Networks (TCNs) (Bai et al., 2018) for sequence modeling tasks across neural machine translations (Vaswani et al., 2017) , language understanding (Devlin et al., 2018) , sequence prediction<cite> (Dai et al., 2019)</cite> , image generation (Child et al., 2019) , video activity classification (Wang et al., 2018) , music generation (Huang et al., 2018a) , and multimodal sentiment analysis (Tsai et al., 2019a) .