Sepp Hochreiter Juergen Schmidhuber vanishing gradient problem residual long short-term memory recurrent neural networks Felix Gers Schmidhuber LSTM LSTM Highway network feedforward neural network ImageNet Highway network Residual neural network recurrent neural network machine learning deep learning neural networks
Sepp Hochreiter's diploma thesis (1991) was called "one of the most important documents in the history of machine learning" by his supervisor Juergen Schmidhuber. Hochreiter identified and analyzed the vanishing gradient problem and proposed recurrent residual connections to solve it. This led to the deep learning method called long short-term memory (LSTM), published in Neural Computation (1997). LSTM recurrent neural networks can learn "very deep learning" tasks with long credit assignment paths that require memories of events that happened thousands of discrete time steps before. The "vanilla LSTM" with forget gate was introduced in 1999 by Felix Gers, Schmidhuber and Fred Cummins. LSTM has become the most cited neural network of the 20th century.
In 2015, Rupesh Kumar Srivastava, Klaus Greff, and Schmidhuber used the LSTM principle to create the Highway network, a feedforward neural network with hundreds of layers, much deeper than previous networks. 7 months later, Kaiming He, Xiangyu Zhang;  Shaoqing Ren, and Jian Sun won the ImageNet 2015 competition with an open-gated or gateless Highway network variant called Residual neural network. This has become the most cited neural network of the 21st century.
