The machine learning models have been limited by how far back the impact of a specific data sample can be considered during the learning. In some cases, the auto-regressive nature of the training of machine learning models leads to memorization of past observations rather than the generalization of the training examples to new data [1]}, [2]}. Transformers address these challenges by using self-attention and positional encoding techniques to jointly attend to and encode the order information as they analyze current data samples in the sequence. These techniques keep the sequential information intact for learning while getting rid of the classical notion of recurrence [3]}. These techniques further allow Transformers to exploit parallelism offered by GPUs and TPUs. Recently, there have been some research attempts to incorporate recurrent components in Transformers [4]}.
