Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> We present a method to learn and recognize object class models from unlabeled and unsegmented cluttered scenes in a scale invariant manner. Objects are modeled as flexible constellations of parts. A probabilistic representation is used for all aspects of the object: shape, appearance, occlusion and relative scale. An entropy-based feature detector is used to select regions and their scale within the image. In learning the parameters of the scale-invariant object model are estimated. This is done using expectation-maximization in a maximum-likelihood setting. In recognition, this model is used in a Bayesian manner to classify images. The flexible nature of the model is demonstrated by excellent results over a range of datasets including geometrically constrained classes (e.g. faces, cars) and flexible objects (such as animals). <s> BIB001 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> We present a novel method for generic visual categorization: the problem of identifying the object content of natural images while generalizing across variations inherent to the object class. This bag of keypoints method is based on vector quantization of affine invariant descriptors of image patches. We propose and compare two alternative implementations using different classifiers: Naive Bayes and SVM. The main advantages of the method are that it is simple, computationally efficient and intrinsically invariant. We present results for simultaneously classifying seven semantic visual categories. These results clearly demonstrate that the method is robust to background clutter and produces good categorization accuracy even without exploiting geometric information. <s> BIB002 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> We propose a novel approach to learn and recognize natural scene categories. Unlike previous work, it does not require experts to annotate the training set. We represent the image of a scene by a collection of local regions, denoted as codewords obtained by unsupervised learning. Each region is represented as part of a "theme". In previous work, such themes were learnt from hand-annotations of experts, while our method learns the theme distributions as well as the codewords distribution over the themes without supervision. We report satisfactory categorization performances on a large set of 13 categories of complex scenes. <s> BIB003 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> Recently SVMs using spatial pyramid matching (SPM) kernel have been highly successful in image classification. Despite its popularity, these nonlinear SVMs have a complexity O(n2 ~ n3) in training and O(n) in testing, where n is the training size, implying that it is nontrivial to scaleup the algorithms to handle more than thousands of training images. In this paper we develop an extension of the SPM method, by generalizing vector quantization to sparse coding followed by multi-scale spatial max pooling, and propose a linear SPM kernel based on SIFT sparse codes. This new approach remarkably reduces the complexity of SVMs to O(n) in training and a constant in testing. In a number of image categorization experiments, we find that, in terms of classification accuracy, the suggested linear SPM based on sparse coding of SIFT descriptors always significantly outperforms the linear SPM kernel on histograms, and is even better than the nonlinear SPM kernels, leading to state-of-the-art performance on several benchmarks by using a single type of descriptors. <s> BIB004 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. To make training faster, we used non-saturating neurons and a very efficient GPU implementation of the convolution operation. To reduce overriding in the fully-connected layers we employed a recently-developed regularization method called "dropout" that proved to be very effective. We also entered a variant of this model in the ILSVRC-2012 competition and achieved a winning top-5 test error rate of 15.3%, compared to 26.2% achieved by the second-best entry. <s> BIB005 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> In this paper, we propose a multi-manifold deep metric learning (MMDML) method for image set classification, which aims to recognize an object of interest from a set of image instances captured from varying viewpoints or under varying illuminations. Motivated by the fact that manifold can be effectively used to model the nonlinearity of samples in each image set and deep learning has demonstrated superb capability to model the nonlinearity of samples, we propose a MMDML method to learn multiple sets of nonlinear transformations, one set for each object class, to nonlinearly map multiple sets of image instances into a shared feature subspace, under which the manifold margin of different class is maximized, so that both discriminative and class-specific information can be exploited, simultaneously. Our method achieves the state-of-the-art performance on five widely used datasets. <s> BIB006 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> We adopt Convolutional Neural Networks (CNN) as our parametric model to learn discriminative features and classifiers for local patch classification. As visually similar pixels are indistinguishable from local context, we alleviate such ambiguity by introducing a global scene constraint. We estimate the global potential in a non-parametric framework. Furthermore, a large margin based CNN metric learning method is proposed for better global potential estimation. The final pixel class prediction is performed by integrating local and global beliefs. Even without any post-processing, we achieve state-of-the-art performance on SiftFlow and competitive results on Stanford Background benchmark. <s> BIB007 </s> Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey <s> V. IMAGE CLASSIFICATION <s> In image labeling, local representations for image units are usually generated from their surrounding image patches, thus long-range contextual information is not effectively encoded. In this paper, we introduce recurrent neural networks (RNNs) to address this issue. Specifically, directed acyclic graph RNNs (DAG-RNNs) are proposed to process DAG-structured images, which enables the network to model long-range semantic dependencies among image units. Our DAG-RNNs are capable of tremendously enhancing the discriminative power of local representations, which significantly benefits the local classification. Meanwhile, we propose a novel class weighting function that attends to rare classes, which phenomenally boosts the recognition accuracy for non-frequent classes. Integrating with convolution and deconvolution layers, our DAG-RNNs achieve new state-of-the-art results on the challenging SiftFlow, CamVid and Barcelona benchmarks. <s> BIB008
Image classification has been an active research topic in the past few decades. Many methods have been proposed to achieve very promising results by using Bag-of-Words (BoW) representation BIB002 , spatial pyramid matching  , topic models BIB003 , part-based models BIB001 and sparse coding BIB004 , etc. However, these methods utilize raw pixel values or hand-crafted features which cannot capture data-driven representations for specific input data. Recently, deep learning has achieved very promising results in image classification. We review some representative works as follows. Krizhevsky et al. BIB005 develop a deep convolutional neural network which has 60 million parameters and 650, 000 neurons. The deep model includes five convolutional layers followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax. The architecture of the deep convolutional neural network proposed by Krizhevsky et al. BIB005 is illustrated in Figure 13 . The deep model achieved very impressive results on the ImageNet LSVRC-2010 contest by reducing the error rates down to 8% against the then-stateof-the-art. The dataset includes 1.2 million high-resolution images in 1000 different classes. This competition has become one of the largest and most challenging computer vision challenges and is held annually. The challenge has attracted not only academic research groups but also industry companies. For example, Google has won the classification challenge in the ImageNet 2014 with a error rate 6.66%. Nowadays, high performance computing plays a very important role in deep learning. Recently, the Chinese search engine company Baidu has obtained a 5.98% error rate on the ImageNet classification dataset by using a supercomputer called Minwa, which consists of 36 server nodes, each with 4 Nvidia Tesla K40m GPUs. Baidu also claims that Minwa can handle higher-resolution images and the larger training dataset ( 2 billion images) which is generated by distorting, flipping and changing colors of the original 1.2 million images. As a result, the system is able to work on real-world photos. Lu et al. BIB006 present a multi-manifold deep metric learning (MMDML) method for image set classification. First, each image set is modeled as a manifold which is passed into multiple layers of deep neural networks and mapped into another feature space. Specifically, the deep network is classspecific so that different classes have different parameters in their networks. Then, the maximal manifold margin criterion is used to learn the parameters of these manifold. In the testing stage, these class-specific deep networks are applied to compute the similarity between the testing image set and all training classes. Finally, the smallest distance is used for classification. As a result, the proposed method can achieve the state-of-the-art performance on five widely used datasets by exploiting both discriminative and class-specific information. Zuo et al.  develop an end-to-end Convolutional Hierarchical Recurrent Neural Networks (C-HRNNs) to explore contextual dependencies for image classification. Figure 15 illustrates the overall framework of C-HRNNs. First, midlevel representations for image regions are extracted by using five layers of Convolutional Neural Networks (CNNs). Then, the output of the fifth CNN layer is pooled into multiple scales. For each scale, spatial dependencies are captured by direct or indirect connections between each region and its surrounding neighbors. For different scales, scale dependencies are encoded by transferring information from the higher Fig. 16 . Illustration of the framework of the approach BIB007 . Fig. 17 . Illustration of the architecture of the full labeling network in the approach BIB008 . level scales to corresponding areas at the lower level scales. Finally, different scale HRNN outputs are collected and put into two fully connected layers. C-HRNNs not only make use of the representation power of CNNs, but also efficiently encodes spatial and scale dependencies among different image regions. As a result, the proposed model achieves state-ofthe-art performance on four challenging image classification benchmarks.