Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> PCT No. PCT/PT98/00006 Sec. 371 Date Jun. 8, 1999 Sec. 102(e) Date Jun. 8, 1999 PCT Filed Dec. 2, 1998 PCT Pub. No. WO99/28676 PCT Pub. Date Jun. 10, 1999Device for protecting an ejection outlet of a burner mounted through a wall of a furnace, comprising a peripheral heat shield around the ejection outlet of the burner comprising a consumable structure comprising a refractory material and a mounting for heat shield comprising a mover which moves the heat shield relative to the wall of the furnace, between at least two positions which are spaced apart along the axis of the burner. <s> BIB001 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> The ability of learning networks to generalize can be greatly enhanced by providing constraints from the task domain. This paper demonstrates how such constraints can be integrated into a backpropagation network through the architecture of the network. This approach has been successfully applied to the recognition of handwritten zip code digits provided by the U.S. Postal Service. A single network learns the entire recognition operation, going from the normalized image of the character to the final classification. <s> BIB002 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> Multilayer neural networks trained with the back-propagation algorithm constitute the best example of a successful gradient based learning technique. Given an appropriate network architecture, gradient-based learning algorithms can be used to synthesize a complex decision surface that can classify high-dimensional patterns, such as handwritten characters, with minimal preprocessing. This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task. Convolutional neural networks, which are specifically designed to deal with the variability of 2D shapes, are shown to outperform all other techniques. Real-life document recognition systems are composed of multiple modules including field extraction, segmentation recognition, and language modeling. A new learning paradigm, called graph transformer networks (GTN), allows such multimodule systems to be trained globally using gradient-based methods so as to minimize an overall performance measure. Two systems for online handwriting recognition are described. Experiments demonstrate the advantage of global training, and the flexibility of graph transformer networks. A graph transformer network for reading a bank cheque is also described. It uses convolutional neural network character recognizers combined with global training techniques to provide record accuracy on business and personal cheques. It is deployed commercially and reads several million cheques per day. <s> BIB003 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> The support-vector network is a new learning machine for two-group classification problems. The machine conceptually implements the following idea: input vectors are non-linearly mapped to a very high-dimension feature space. In this feature space a linear decision surface is constructed. Special properties of the decision surface ensures high generalization ability of the learning machine. The idea behind the support-vector network was previously implemented for the restricted case where the training data can be separated without errors. We here extend this result to non-separable training data. ::: ::: High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated. We also compare the performance of the support-vector network to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition. <s> BIB004 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> This work contains a theoretical study and computer simulations of a new self-organizing process. The principal discovery is that in a simple network of adaptive physical elements which receives signals from a primary event space, the signal representations are automatically mapped onto a set of output responses in such a way that the responses acquire the same topological order as that of the primary events. In other words, a principle has been discovered which facilitates the automatic formation of topologically correct maps of features of observable events. The basic self-organizing system is a one- or two-dimensional array of processing units resembling a network of threshold-logic units, and characterized by short-range lateral feedback between neighbouring units. Several types of computer simulations are used to demonstrate the ordering process as well as the conditions under which it fails. <s> BIB005 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> A neural network model for a mechanism of visual pattern recognition is proposed in this paper. The network is self-organized by “learning without a teacher”, and acquires an ability to recognize stimulus patterns based on the geometrical similarity (Gestalt) of their shapes without affected by their positions. This network is given a nickname “neocognitron”. After completion of self-organization, the network has a structure similar to the hierarchy model of the visual nervous system proposed by Hubel and Wiesel. The network consits of an input layer (photoreceptor array) followed by a cascade connection of a number of modular structures, each of which is composed of two layers of cells connected in a cascade. The first layer of each module consists of “S-cells”, which show characteristics similar to simple cells or lower order hypercomplex cells, and the second layer consists of “C-cells” similar to complex cells or higher order hypercomplex cells. The afferent synapses to each S-cell have plasticity and are modifiable. The network has an ability of unsupervised learning: We do not need any “teacher” during the process of self-organization, and it is only needed to present a set of stimulus patterns repeatedly to the input layer of the network. The network has been simulated on a digital computer. After repetitive presentation of a set of stimulus patterns, each stimulus pattern has become to elicit an output only from one of the C-cell of the last layer, and conversely, this C-cell has become selectively responsive only to that stimulus pattern. That is, none of the C-cells of the last layer responds to more than one stimulus pattern. The response of the C-cells of the last layer is not affected by the pattern's position at all. Neither is it affected by a small change in shape nor in size of the stimulus pattern. <s> BIB006 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> To achieve its impressive performance in tasks such as speech perception or object recognition, the brain extracts multiple levels of representation from the sensory input. Backpropagation was the first computationally efficient model of how neural networks could learn multiple layers of representation, but it required labeled training data and it did not work well in deep networks. The limitations of backpropagation learning can now be overcome by using multilayer neural networks that contain top-down connections and training them to generate sensory data rather than to classify it. Learning multilayer generative models might seem difficult, but a recent discovery makes it easy to learn nonlinear distributed representations one layer at a time. <s> BIB007 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> Sparse coding provides a class of algorithms for finding succinct representations of stimuli; given only unlabeled input data, it discovers basis functions that capture higher-level features in the data. However, finding sparse codes remains a very difficult computational problem. In this paper, we present efficient sparse coding algorithms that are based on iteratively solving two convex optimization problems: an L1-regularized least squares problem and an L2-constrained least squares problem. We propose novel algorithms to solve both of these optimization problems. Our algorithms result in a significant speedup for sparse coding, allowing us to learn larger sparse codes than possible with previously described algorithms. We apply these algorithms to natural images and demonstrate that the inferred sparse codes exhibit end-stopping and non-classical receptive field surround suppression and, therefore, may provide a partial explanation for these two phenomena in V1 neurons. <s> BIB008 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in state-of-the-art-performance. <s> BIB009 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> We present a new learning algorithm for Boltzmann machines that contain many layers of hidden variables. Data-dependent expectations are estimated using a variational approximation that tends to focus on a single mode, and dataindependent expectations are approximated using persistent Markov chains. The use of two quite different techniques for estimating the two types of expectation that enter into the gradient of the log-likelihood makes it practical to learn Boltzmann machines with multiple hidden layers and millions of parameters. The learning can be made more efficient by using a layer-by-layer “pre-training” phase that allows variational inference to be initialized with a single bottomup pass. We present results on the MNIST and NORB datasets showing that deep Boltzmann machines learn good generative models and perform well on handwritten digit and visual object recognition tasks. <s> BIB010 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> The promise of unsupervised learning methods lies in their potential to use vast amounts of unlabeled data to learn complex, highly nonlinear models with millions of free parameters. We consider two well-known unsupervised learning models, deep belief networks (DBNs) and sparse coding, that have recently been applied to a flurry of machine learning applications (Hinton & Salakhutdinov, 2006; Raina et al., 2007). Unfortunately, current learning algorithms for both models are too slow for large-scale applications, forcing researchers to focus on smaller-scale models, or to use fewer training examples. In this paper, we suggest massively parallel methods to help resolve these problems. We argue that modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods. We develop general principles for massively parallelizing unsupervised learning tasks using graphics processors. We show that these principles can be applied to successfully scaling up learning algorithms for both DBNs and sparse coding. Our implementation of DBN learning is up to 70 times faster than a dual-core CPU implementation for large models. For example, we are able to reduce the time required to learn a four-layer DBN with 100 million free parameters from several weeks to around a single day. For sparse coding, we develop a simple, inherently parallel algorithm, that leads to a 5 to 15-fold speedup over previous methods. <s> BIB011 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> We explore an original strategy for building deep networks, based on stacking layers of denoising autoencoders which are trained locally to denoise corrupted versions of their inputs. The resulting algorithm is a straightforward variation on the stacking of ordinary autoencoders. It is however shown on a benchmark of classification problems to yield significantly lower classification error, thus bridging the performance gap with deep belief networks (DBN), and in several cases surpassing it. Higher level representations learnt in this purely unsupervised fashion also help boost the performance of subsequent SVM classifiers. Qualitative experiments show that, contrary to ordinary autoencoders, denoising autoencoders are able to learn Gabor-like edge detectors from natural image patches and larger stroke detectors from digit images. This work clearly establishes the value of using a denoising criterion as a tractable unsupervised objective to guide the learning of useful higher level representations. <s> BIB012 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB013 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> Pedestrian detection is a problem of considerable practical interest. Adding to the list of successful applications of deep learning methods to vision, we report state-of-the-art and competitive results on all major pedestrian datasets with a convolutional network model. The model uses a few new twists, such as multi-stage features, connections that skip layers to integrate global shape information with local distinctive motif information, and an unsupervised method based on convolutional sparse coding to pre-train the filters at each stage. <s> BIB014 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> We develop and present a novel deep convolutional neural network architecture, where heterogeneous pooling is used to provide constrained frequency-shift invariance in the speech spectrogram while minimizing speech-class confusion induced by such invariance. The design of the pooling layer is guided by domain knowledge about how speech classes would change when formant frequencies are modified. The convolution and heterogeneous-pooling layers are followed by a fully connected multi-layer neural network to form a deep architecture interfaced to an HMM for continuous speech recognition. During training, all layers of this entire deep net are regularized using a variant of the “dropout” technique. Experimental evaluation demonstrates the effectiveness of both heterogeneous pooling and dropout regularization. On the TIMIT phonetic recognition task, we have achieved an 18.7% phone error rate, lowest on this standard task reported in the literature with a single system and with no use of information about speaker identity. Preliminary experiments on large vocabulary speech recognition in a voice search task also show error rate reduction using heterogeneous pooling in the deep convolutional neural network. <s> BIB015 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> In modern face recognition, the conventional pipeline consists of four stages: detect => align => represent => classify. We revisit both the alignment step and the representation step by employing explicit 3D face modeling in order to apply a piecewise affine transformation, and derive a face representation from a nine-layer deep neural network. This deep network involves more than 120 million parameters using several locally connected layers without weight sharing, rather than the standard convolutional layers. Thus we trained it on the largest facial dataset to-date, an identity labeled dataset of four million facial images belonging to more than 4, 000 identities. The learned representations coupling the accurate model-based alignment with the large facial database generalize remarkably well to faces in unconstrained environments, even with a simple classifier. Our method reaches an accuracy of 97.35% on the Labeled Faces in the Wild (LFW) dataset, reducing the error of the current state of the art by more than 27%, closely approaching human-level performance. <s> BIB016 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> Defining hand-crafted feature representations needs expert knowledge, requires timeconsuming manual adjustments, and besides, it is arguably one of the limiting factors of object tracking. In this paper, we propose a novel solution to automatically relearn the most useful feature representations during the tracking process in order to accurately adapt appearance changes, pose and scale variations while preventing from drift and tracking failures. We employ a candidate pool of multiple Convolutional Neural Networks (CNNs) as a data-driven model of different instances of the target object. Individually, each CNN maintains a specific set of kernels that favourably discriminate object patches from their surrounding background using all available low-level cues. These kernels are updated in an online manner at each frame after being trained with just one instance at the initialization of the corresponding CNN. Given a frame, the most promising CNNs in the pool are selected to evaluate the hypothesises for the target object. The hypothesis with the highest score is assigned as the current detection window and the selected models are retrained using a warm-start back-propagation which optimizes a structural loss function. In addition to the model-free tracker, we introduce a class-specific version of the proposed method that is tailored for tracking of a particular object class such as human faces. Our experiments on a large selection of videos from the recent benchmarks demonstrate that our method outperforms the existing state-of-the-art algorithms and rarely loses the track of the target object. <s> BIB017 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> We adopt Convolutional Neural Networks (CNN) as our parametric model to learn discriminative features and classifiers for local patch classification. As visually similar pixels are indistinguishable from local context, we alleviate such ambiguity by introducing a global scene constraint. We estimate the global potential in a non-parametric framework. Furthermore, a large margin based CNN metric learning method is proposed for better global potential estimation. The final pixel class prediction is performed by integrating local and global beliefs. Even without any post-processing, we achieve state-of-the-art performance on SiftFlow and competitive results on Stanford Background benchmark. <s> BIB018 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> Most existing feature learning-based methods for RGB-D object recognition either combine RGB and depth data in an undifferentiated manner from the outset, or learn features from color and depth separately, which do not adequately exploit different characteristics of the two modalities or utilize the shared relationship between the modalities. In this paper, we propose a general CNN-based multi-modal learning framework for RGB-D object recognition. We first construct deep CNN layers for color and depth separately, which are then connected with a carefully designed multi-modal layer. This layer is designed to not only discover the most discriminative features for each modality, but is also able to harness the complementary relationship between the two modalities. The results of the multi-modal layer are back-propagated to update parameters of the CNN layers, and the multi-modal feature learning and the back-propagation are iteratively performed until convergence. Experimental results on two widely used RGB-D object datasets show that our method for general multi-modal learning achieves comparable performance to state-of-the-art methods specifically designed for RGB-D data. <s> BIB019 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> We adopt Convolutional Neural Networks (CNN) to learn discriminative features for local patch classification. We further introduce quaddirectional 2D Recurrent Neural Networks to model the long range dependencies among pixels. Our quaddirectional 2D-RNN is able to embed the global image context into the compact local representation, which significantly enhance their discriminative power. Our experiments demonstrate that the integration of CNN and quaddirectional 2D-RNN achieves very promising results which are comparable to state-of-the-art on real-world image labeling benchmarks. <s> BIB020 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> In order to encode the class correlation and class specific information in image representation, we propose a new local feature learning approach named Deep Discriminative and Shareable Feature Learning (DDSFL). DDSFL aims to hierarchically learn feature transformation filter banks to transform raw pixel image patches to features. The learned filter banks are expected to (1) encode common visual patterns of a flexible number of categories; (2) encode discriminative information; and (3) hierarchically extract patterns at different visual levels. Particularly, in each single layer of DDSFL, shareable filters are jointly learned for classes which share the similar patterns. Discriminative power of the filters is achieved by enforcing the features from the same category to be close, while features from different categories to be far away from each other. Furthermore, we also propose two exemplar selection methods to iteratively select training data for more efficient and effective learning. Based on the experimental results, DDSFL can achieve very promising performance, and it also shows great complementary effect to the state-of-the-art Caffe features. HighlightsWe propose to encode shareable and discriminative information in feature learning.Two exemplar selection methods are proposed to select effective training data.We build a hierarchical learning scheme to capture multiple visual level information.Our DDSFL outperforms most of the existing features.DDSFL features show great complementary effect to Caffe features. <s> BIB021 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> This paper proposes a joint multi-task learning algorithm to better predict attributes in images using deep convolutional neural networks (CNN). We consider learning binary semantic attributes through a multi-task CNN model, where each CNN will predict one binary attribute. The multi-task learning allows CNN models to simultaneously share visual knowledge among different attribute categories. Each CNN will generate attribute-specific feature representations, and then we apply multi-task learning on the features to predict their attributes. In our multi-task framework, we propose a method to decompose the overall model’s parameters into a latent task matrix and combination matrix. Furthermore, under-sampled classifiers can leverage shared statistics from other classifiers to improve their performance. Natural grouping of attributes is applied such that attributes in the same group are encouraged to share more knowledge. Meanwhile, attributes in different groups will generally compete with each other, and consequently share less knowledge. We show the effectiveness of our method on two popular attribute datasets. <s> BIB022 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> I. INTRODUCTION <s> Matching people across multiple camera views known as person reidentification is a challenging problem due to the change in visual appearance caused by varying lighting conditions. The perceived color of the subject appears to be different under different illuminations. Previous works use color as it is or address these challenges by designing color spaces focusing on a specific cue. In this paper, we propose an approach for learning color patterns from pixels sampled from images across two camera views. The intuition behind this work is that, even though varying lighting conditions across views affect the pixel values of the same color, the final representation of a particular color should be stable and invariant to these variations, i.e., they should be encoded with the same values. We model color feature generation as a learning problem by jointly learning a linear transformation and a dictionary to encode pixel values. We also analyze different photometric invariant color spaces as well as popular color constancy algorithm for person reidentification. Using color as the only cue, we compare our approach with all the photometric invariant color spaces and show superior performance over all of them. Combining with other learned low-level and high-level features, we obtain promising results in VIPeR, Person Re-ID 2011, and CAVIAR4REID data sets. <s> BIB023
A smart city aims to improve quality and performance of urban services by using digital technologies or information and communication technologies. Data analytics plays an important role in smart cities. Many sensors are installed in a smart city to capture a huge volume of data such as surveillance videos, environment and transportation data. To capture useful information from such big data, machine learning algorithms are often used and have achieved very promising results in a wide range of applications, e.g. video analytics. Therefore, leveraging on machine learning can facilitate smart city development. Machine learning aims to develop the computer algorithms which can learn experience from example inputs and make data-driven predictions on unknown test data. Such algorithms can be divided into two categories: supervised learning (e.g. BIB004  ) and unsupervised learning (e.g. BIB005  ). Given labeled input and output pairs, supervised learning aims to find a mapping rule for predicting outputs of unknown inputs. In contrast, unsupervised learning focuses on exploring intrinsic characteristics of inputs. Supervised learning and unsupervised learning are complementary to each other. Since supervised learning leverages labels of inputs which are meaningful to human, it is easy to apply this kind of learning algorithms to pattern classification and data regression problems. However, supervised learning relies on labeled data which could cost a lot of manual works. Moreover, there are uncertainties and ambiguities in labels. In other words, the label for an object is not unique. To mitigate these problems, unsupervised learning can be used to handle intra-class variation as it does not require labels of data. In the past decades, machine learning methods have been applied to a wide range of applications such as bioinformatics, computer vision, medical diagnosis, natural language processing, robotics, sentiment analysis, speech L. Wang and D. Sng are with the Rapid-Rich Object Search (ROSE) Lab at the Nanyang Technological University, Singapore 637553 (e-mail: wa0002li@e.ntu.edu.sg; dennis.sng@ntu.edu.sg). Fig. 1 . Illustration of the CNN proposed by LeCun et al. BIB003 for digits recognition. recognition and stock market analysis, etc. In this paper, we focus on reviewing the recent achievements of deep learning which is a subfield of machine learning. In contrast with shallow learning algorithms, deep learning aims to extract hierarchical representations from large-scale data (e.g. images and videos) by using deep architecture models with multiple layers of non-linear transformations. With such learned feature representations, it becomes easier to achieve better performance than using raw pixel values or hand-crafted features. The principle behind this success is that deep learning is able to disentangle different levels of abstractions embedded in observed data by elaborately designing the layer depth and width, and properly selecting features that are beneficial for learning tasks. In fact, the history of deep learning starts at least from 1980, when Neocognitron BIB006 is proposed by Fukushima. In 1989, LeCun et al. BIB002 propose to apply backpropagation onto a deep neural network for handwritten ZIP code recognition. However, the training time on the network was too long for practical use. Also, deep neural networks have been studied in speech recognition for many years, but can hardly surpass the shallow generative models. It is due to the fact that deep learning architectures require large training data which was scarce in those early days. Hinton et al.  review these difficulties and claim their confidence of solving these issues for applying deep learning to speech recognition, since Hinton BIB007 has achieved breakthroughs on training multilayer neural networks by pre-training one layer at a time as an unsupervised restricted Boltzmann machine and then using supervised backpropagation for fine-tuning. Since the breakthrough of deep learning, it has been applied to many other research areas besides speech recognition. Deep learning architectures have different variants such as Deep Belief Networks (DBN) BIB001 , Convolutional Neural Networks (CNN) BIB013 , Deep Boltzmann Machines (DBM) BIB010 and Stacked Denoising Auto-Encoders (SDAE) BIB012 , etc. The most attractive model is Convolutional Neural Networks which have achieved very promising results in both computer vision and speech recognition. An illustration of the CNN proposed by LeCun et al. BIB003 for digits recognition is presented in Figure 1 . As shown in the figure, a CNN usually consists of convolutional layers, pooling layers and fully connected layers. With loss layers on the top of the CNN, the whole network can be trained end-to-end by using the backpropagation algorithm. Compared to other deep feed-forward neural networks, a CNN is easier to train as it has fewer parameters to estimate. As a result, CNN has a wide range of applications such as image classification  , face recognition BIB016 , object tracking BIB017 , pedestrian detection BIB014 , attribute prediction BIB022 , scene labeling BIB018 , person re-identification BIB023 , RGB-D object recognition BIB019 , image labeling BIB020 , scene image classification BIB021 , speech recognition BIB015 and natural language processing BIB009 , etc. Deep learning has received much attention from not only academic but also industry. For example, Geoff Hinton and Li Deng started their collaborations from 2009 in the focus of applying deep learning to large-scale speech recognition, in which the performance is significantly improved against the traditional generative models by using big training data and the correspondingly designed deep neural networks. Another exciting example is that Andrew Ng and Jeff Dean from the Google Brain team successfully extract object-level semantics (e.g. cats) from unlabeled YouTube videos by using a neural network with the self-taught capacity. In future, deep learning will have more and more real applications in industry. Besides methodology breakthroughs and available big training data, the recent success for deep learning is also due to advances in hardware. Specifically, an electronic circuit called Graphics Processor Unit (GPU) is designed for accelerating the algorithms that need to process a large number of blocks of data, and is characteristic of its highly parallel structure. For example, Nvidia's latest GPU, the GTX 980, is based on their 10th generation GPU architecture, called Maxwell, which delivers double the performance per watt compared to the previous generation. The GM 204 chip is composed of an array of 4 Graphics Processing Clusters (GPCs), 16 Streaming Multiprocessors (SMs), and 4 memory controllers. The GeForce GTX 980 uses the full complement of these architectural components. Its parallel structure is illustrated in Figure 2 . It is necessary to mention that the Nvidia company has made a lot of contributions for popularizing GPUs, e.g. Nvidia marketed the GeForce 256 as the world's first GPU. Recently, GPUs are very popular in machine learning. For a general instance, Raina et al. BIB011 propose to accelerate the sparse coding algorithm BIB008 by using GPUs and finally speed up the previous method using a dual-core CPU up to 15 times. In particular, Raina et al. BIB011 also report that the GPU speedup on learning Deep Belief Networks (DBNs) achieves up to 70 times against a dual-core CPU implementation. For learning a four-layer DBN with 100 million parameters, using GPU rather than CPU is able to reduce the required time from several weeks to around a single day. Smart city is so-called "smart" as it has the capability of computing and analyzing urban data collected from e.g. monitoring systems, government agencies, commercial companies and social networking websites. Since deep learning is suitable for handling large-scale data, it can be used to process and analyze millions of video data captured from the distributed sensors in a smart city. Regarding such data, there are many active research topics such as object detection, object tracking, face recognition, image classification and scene labeling. In the following sections, we review the state-of-the-art deep learning algorithms in these application areas.