My First Deep Learning System of 1991 + Deep Learning Timeline 1962-2013

Deep Learning has attracted significant attention in recent years. Here I present a brief overview of my first Deep Learner of 1991, and its historic context, with a timeline of Deep Learning highlights.

: My first Deep Learning system of 1991 used a deep stack of recurrent neural networks (a Neural Hierarchical Temporal Memory) pre-trained in unsupervised fashion to accelerate subsequent supervised learning [79,81,82].
In 2009, our Deep Learning Artificial Neural Networks became the first Deep Learners to win official international pattern recognition competitions [40,83] (with secret test sets known only to the organisers); by 2012 they had won eight of them (TL 1.13), including the first contests on object detection in large images (ICPR 2012) [2,16] and image segmentation (ISBI 2012) [3,15]. In 2011, they achieved the world's first superhuman visual pattern recognition results [20,19]. Others have implemented very similar techniques, e.g., [51], and won additional contests or set benchmark records since 2012, e.g., (TL 1.13, TL 1.14). The field of Deep Learning research is far older though-compare the timeline (TL) further down.
My first Deep Learner dates back to 1991 [79,81,82] (TL 1.7). It can perform credit assignment across hundreds of nonlinear operators or neural layers, by using unsupervised pre-training for a stack of recurrent neural networks (RNN) (deep by nature) as in Figure 1. Such RNN are general computers more powerful than normal feedforward NN, and can encode entire sequences of input vectors.
The basic idea is still relevant today. Each RNN is trained for a while in unsupervised fashion to predict its next input. From then on, only unexpected inputs (errors) convey new information and get fed to the next higher RNN which thus ticks on a slower, self-organising time scale. It can easily be shown that no information gets lost. It just gets compressed (much of machine learning is essentially about compression). We get less and less redundant input sequence encodings in deeper and deeper levels of this hierarchical temporal memory, which compresses data in both space (like feedforward NN) and time. There also is a continuous variant [84].
One ancient illustrative Deep Learning experiment of 1993 [82] required credit assignment across 1200 time steps, or through 1200 subsequent nonlinear virtual layers. The top level code of the initially unsupervised RNN stack, however, got so compact that (previously infeasible) sequence classification through additional supervised learning became possible.
There is a way of compressing higher levels down into lower levels, thus partially collapsing the hierarchical temporal memory. The trick is to retrain lower-level RNN to continually imitate (predict) the hidden units of already trained, slower, higher-level RNN, through additional predictive output neurons [81,79,82]. This helps the lower RNN to develop appropriate, rarely changing memories that may bridge very long time lags.
The Deep Learner of 1991 was a first way of overcoming the Fundamental Deep Learning Problem identified and analysed in 1991 by my very first student (now professor) Sepp Hochreiter (TL 1.6): the problem of vanishing or exploding gradients [46,47,10]. The latter motivated all our subsequent Deep Learning research of the 1990s and 2000s.
Particularly successful are stacks of LSTM RNN [40] trained by Connectionist Temporal Classification (CTC) [36]. On faster computers of 2009, this became the first RNN system ever to win an official international pattern recognition competition [40,83], through the work of my PhD student and postdoc Alex Graves, e.g., [40]. To my knowledge, this also was the first Deep Learning system ever (recurrent or not) to win such a contest (TL 1.10). (In fact, it won three different ICDAR 2009 contests on connected handwriting in three different languages, e.g., [83,40], TL 1.10.) A while ago, Alex moved on to Geoffrey Hinton's lab (Univ. Toronto), where a stack [40] of our bidirectional LSTM RNN [39] also broke a famous TIMIT speech recognition record [38] (TL 1.14), despite thousands of man years previously spent on HMM-based speech recognition research. CTC-LSTM also helped to score first at NIST's OpenHaRT2013 evaluation [11].
Our (multi-column [19]) GPU-MPCNN [18] (TL 1.12) were adopted by the groups of Univ. Toronto / Stanford / Google, e.g., [51,25] (TL 1.13, TL 1.14). Apple Inc., the most profitable smartphone maker, hired Ueli Meier, member of our Deep Learning team that won the ICDAR 2011 Chinese handwriting contest [22]. ArcelorMittal, the world's top steel producer, is using our methods for material defect detection and classification, e.g., [63]. Other users include a leading automotive supplier, recent start-ups such as deepmind (which hired four of my former PhD students/postdocs), and many other companies and leading research labs. One of the most important applications of our techniques is biomedical imaging [16] (TL 1.13, TL 1.14), e.g., for cancer prognosis or plaque detection in CT heart scans.
In particular, most competition-winning or benchmark record-setting Deep Learners (TL 1.10 -TL 1.14) use one of two supervised techniques developed in my lab: (1) recurrent LSTM (1997) (TL 1.8) trained by CTC (2006) [36], or (2) feedforward GPU-MPCNN (2011) [18] (TL 1.12) (building on earlier work since the 1960s mentioned in the text above). Nevertheless, in many applications it can still be advantageous to combine the best of both worlds -supervised learning and unsupervised pre-training, like in my 1991 system described above [79,81,82].

1991: Fundamental Deep Learning Problem
By the early 1990s, experiments had shown that deep feedforward or recurrent networks are hard to train by backpropagation (TL 1.2). My student Hochreiter [46] discovered and analyzed the reason, namely, the Fundamental Deep Learning Problem due to vanishing or exploding gradients. Compare [47].

1991: Deep Hierarchy of Recurrent NN
My first recurrent Deep Learning system (present paper) partially overcame the fundamental problem (TL 1.6) through a deep RNN stack pre-trained in unsupervised fashion [79,81,82] to accelerate subsequent supervised learning. This was a working Deep Learner in the modern post-2000 sense, and also the first Neural Hierarchical Temporal Memory.

2010: Plain Backpropagation on GPUs Yields Excellent Results
In 2010, a new MNIST record was set by good old backpropagation (TL 1.2) in deep but otherwise standard NN, without unsupervised pre-training, and without convolution (but with training pattern deformations). This was made possible mainly by boosting computing power through a fast GPU implementation [17]. (A year later, first human-competitive performance on MNIST was achieved by a deep MCMPCNN (TL 1.12) [22].)

2011: MPCNN on GPU / First Superhuman Visual Pattern Recognition
In 2011, Ciresan et al. introduced supervised GPU-based Max-Pooling CNN or Convnets (MPCNN) [18], today used by most if not all feedforward competition-winning deep NN (TL 1.13, TL 1.14). The first superhuman visual pattern recognition in a controlled competition (traffic signs [87]) was achieved [20,19] (twice better than humans, three times better than the closest artificial NN competitor, six times better than the best non-neural method), through deep and wide Multi-Column (MC) GPU-MPCNN [18,19], the current gold standard for deep feedforward NN.

2012: First Contests Won on Object Detection and Image Segmentation
2012 saw the first Deep learning system (a GPU-MCMPCNN [18,19], TL 1.12) to win a contest on visual object detection in large images (as opposed to mere recognition/classification): the ICPR 2012 Contest on Mitosis Detection in Breast Cancer Histological Images [2,74,16]. An MC (TL 1.12) variant of a GPU-MPCNN also achieved best results on the ImageNet classification benchmark [51]. 2012 also saw the first pure image segmentation contest won by Deep Learning (again through a GPU-MCMPCNN), namely, the ISBI 2012 Challenge on segmenting neuronal structures [3,15]. This was the 8th international pattern recognition contest won by my team since 2009 [1].

2013: More Contests and Benchmark Records
In 2013, a new TIMIT phoneme recognition record was set by deep LSTM RNN [38] (TL 1.8, TL 1.10). A new record [24] on the ICDAR Chinese handwriting recognition benchmark (over 3700 classes) was set on a desktop machine by a GPU-MCMPCNN with almost human performance. The MICCAI 2013 Grand Challenge on Mitosis Detection was won by a GPU-MCMPCNN [88,16]. Deep GPU-MPCNN [18] also helped to achieve new best results on ImageNet classification [95] and PASCAL object detection [34].
Additional contests are mentioned in the web pages of the Swiss AI Lab IDSIA, the University of Toronto, NY University, and the University of Montreal.