[1]} solved the vanishing gradient problem in RNNs by introducing control gates in the network. They used three control gates as forget gate, input gate, and output gate. The forget gate (\(\mathbf {f}_t\) ) is used to control how much past information to forget. The input gate (\(\mathbf {i}_t\) ) is used to control how much current information has to be kept. The output gate (\(\mathbf {o}_t\) ) is used to control the exposure of the output. They also added a cell state (\(\mathbf {c}_t\) ) in RNN to remember the past information effectively. The following equations summarize the functionality of an LSTM network.
\(& \mathbf {i}_t = \sigma (\mathbf {W}_i \mathbf {x}_t + \mathbf {U}_i \mathbf {h}_{t-1} + \mathbf {b}_i) \nonumber \\& \mathbf {f}_t = \sigma (\mathbf {W}_f \mathbf {x}_t + \mathbf {U}_f \mathbf {h}_{t-1} + \mathbf {b}_f) \nonumber \\& \tilde{\mathbf {c}}_t = \tanh (\mathbf {W}_c \mathbf {x}_t + \mathbf {U}_c \mathbf {h}_{t-1} + \mathbf {b}_c) \nonumber \\& \mathbf {c}_t = \mathbf {i}_t \circ \tilde{\mathbf {c}}_{t} + \mathbf {f}_t \circ \mathbf {c}_{t-1} \nonumber \\& \mathbf {o}_t = \sigma (\mathbf {W}_o \mathbf {x}_t + \mathbf {U}_o \mathbf {h}_{t-1} + \mathbf {b}_o) \nonumber \\& \mathbf {h}_t = \mathbf {o}_t \circ \tanh (\mathbf {c}_t)\) 
