As earlier mentioned, RNNs are dynamic systems [1]} with an internal state at each time step of the classification resulting from the connections between higher layer neurons and the neurons in the lower layers as well as optional self-feedback connection(s).
Initially developed RN networks such as Elman and Jordan networks [2]} had a limitation of looking back in time for more extended time steps due to the issue of vanishing or exploding gradients. Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) were developed to address this issue.
LSTMs usually have three gates, the input, forget, and output gates that learn overtime what information(s) are essential; the input gates determine whether a piece of information is important and usually use simple sigmoid function units with activation range between 0 and 1 to control the signal into the gate. The forget gate helps to decide if a piece of information should be deleted or kept, while the output gate learns how to control access to cell contents and helps to decide which information is worthy of impacting the output of the current time-step.
