The Transformer is at the heart of many recent breakthroughs and developments in the field of sequence-to-sequence models. It has improved the state-of-the-art in many natural language processing tasks such as machine translation [1]}, document generation [2]} as well as syntactic parsing [3]}. The pretrained transformer language models, Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformer (GPT) proved their effectiveness in tasks such as natural language understanding and question answering [4]}, [5]}. Generative models such as the Image Transformer [6]} and Music Transformer [7]} proved to generate convincingly natural images and musical compositions based on human evaluation.
