RNNs have long been explored for sequential applications for decades, often with good results. However, a significant limitation of vanilla RNN models which strictly integrate state information over time is the ¡°vanishing gradient¡± effect [1]}: the ability to back propagate an error signal through a long-range temporal interval becomes increasingly impossible in practice. To mitigate this problem, a class of models with a long-range dependencies learning capability, called Long Short-Term Memory (LSTM), was introduced by Hochreiter and Schmidhuber [2]}. Specifically, LSTM consists of memory cells, with each cell containing units to learn when to forget previous hidden states and when to update hidden states with new information.
