Due to the successive use of the activation function on the hidden state of the RNN (Eq. (2.11)), RNN suffers from the vanishing or exploding gradient problem for long sequences. The exploding gradient problem can be solved by clipping the gradient up to a certain threshold, but the vanishing gradient problem cannot be solved that easily. [1]} proposed long short-term memory (LSTM) and [2]} proposed gated recurrent unit (GRU) to address the vanishing gradient problem for long sequences. We have used the LSTM network extensively in this thesis and will discuss it briefly here.
