Machine learning models with specialized components and architectures for handling the sequential nature of data have been extensively proposed in the literature and used by the community. The most notable of these machine learning models are Recurrent Neural Networks (RNNs) and their popular variants, including Long-Short Term Memory (LSTM) and Gated Recurrent Units (GRU) [1]}, [2]}, [3]}. These models process batches of data sequentially, one sample at a time, and optimize unknown model parameters using the well-known gradient descent algorithm. The gradient information for updating model parameters is calculated using back-propagation through time (BPTT) [4]}. LSTMs and GRUs have been successfully used in many applications. However, they suffer from several limitations due to the sequential processing of input data and the challenges associated with BPTT, especially while processing datasets with long dependencies. The training process of LSTM and GRU models also suffers from vanishing and exploding gradient problems. While processing long sequences, the gradient descent algorithm (using BPTT) may not update the model parameters as the gradient information is lost (either approaches zero or infinity). Additionally, these models generally do not benefit from the parallel computations offered by graphical processing units (GPUs), tensor processing units (TPUs), and other hardware. Certain architectural modifications and training tricks may help LSTMs and GRUs mitigate gradient related problems to a certain extent. However, challenges in learning from long sequences of data with the limited use of parallelization offered by modern hardware impact the effectiveness and efficiency of RNN based models [5]}.
