A Survey on the New Generation of Deep Learning in Image Processing <s> C. OBJECT DETECTION <s> Extreme learning machine (ELM) is an emerging learning algorithm for the generalized single hidden layer feedforward neural networks, of which the hidden node parameters are randomly generated and the output weights are analytically computed. However, due to its shallow architecture, feature learning using ELM may not be effective for natural signals (e.g., images/videos), even with a large number of hidden nodes. To address this issue, in this paper, a new ELM-based hierarchical learning framework is proposed for multilayer perceptron. The proposed architecture is divided into two main components: 1) self-taught feature extraction followed by supervised feature classification and 2) they are bridged by random initialized hidden weights. The novelties of this paper are as follows: 1) unsupervised multilayer encoding is conducted for feature extraction, and an ELM-based sparse autoencoder is developed via l 1 constraint. By doing so, it achieves more compact and meaningful feature representations than the original ELM; 2) by exploiting the advantages of ELM random feature mapping, the hierarchically encoded outputs are randomly projected before final decision making, which leads to a better generalization with faster learning speed; and 3) unlike the greedy layerwise training of deep learning (DL), the hidden layers of the proposed framework are trained in a forward manner. Once the previous layer is established, the weights of the current layer are fixed without fine-tuning. Therefore, it has much better learning efficiency than the DL. Extensive experiments on various widely used classification data sets show that the proposed algorithm achieves better and faster convergence than the existing state-of-the-art hierarchical learning methods. Furthermore, multiple applications in computer vision further confirm the generality and capability of the proposed learning scheme. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> C. OBJECT DETECTION <s> In this paper, a hierarchical deep multi-task learning (HD-MTL) algorithm is developed to support large-scale visual recognition (e.g., recognizing thousands or even tens of thousands of atomic object classes automatically). To achieve more effective accomplishment of the coarse-to-fine tasks for hierarchical visual recognition, multiple sets of deep features are first extracted from the different layers of deep convolutional neural networks (deep CNNs). A visual tree is then learned by assigning the visually-similar atomic object classes with similar learning complexities into the same group, and it can provide a good environment for identifying the inter-related learning tasks automatically. By leveraging the inter-task relatedness (inter-class similarities) to learn more discriminative group-specific deep representations, our deep multi-task learning algorithm can achieve the global optimum easily and obtain more discriminative node classifiers for distinguishing the visually-similar atomic object classes (in the same group) effectively. Our HD-MTL algorithm can control the inter-level error propagation effectively by using an end-to-end approach for jointly learning more representative deep CNNs (for image representation) and more discriminative tree classifier (for large-scale visual recognition) and updating them simultaneously. Our incremental deep learning algorithms can effectively adapt both the deep CNNs and the tree classifier to the new training images and the new object classes. Our experimental results have demonstrated that our HD-MTL algorithm can achieve very competitive results on both the accuracy rates and the computational efficiency for large-scale visual recognition. <s> BIB002 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> C. OBJECT DETECTION <s> This paper proposes a deep learning model to efficiently detect salient regions in videos. It addresses two important issues: 1) deep video saliency model training with the absence of sufficiently large and pixel-wise annotated video data and 2) fast video saliency training and detection. The proposed deep video saliency network consists of two modules, for capturing the spatial and temporal saliency information, respectively. The dynamic saliency model, explicitly incorporating saliency estimates from the static saliency model, directly produces spatiotemporal saliency inference without time-consuming optical flow computation. We further propose a novel data augmentation technique that simulates video training data from existing annotated image data sets, which enables our network to learn diverse saliency information and prevents overfitting with the limited number of training videos. Leveraging our synthetic video data (150K video sequences) and real videos, our deep video saliency model successfully learns both spatial and temporal saliency cues, thus producing accurate spatiotemporal saliency estimate. We advance the state-of-the-art on the densely annotated video segmentation data set (MAE of .06) and the Freiburg-Berkeley Motion Segmentation data set (MAE of .07), and do so with much improved speed (2 fps with all steps). <s> BIB003 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> C. OBJECT DETECTION <s> State-of-the-art object detection networks depend on region proposal algorithms to hypothesize object locations. Advances like SPPnet [1] and Fast R-CNN [2] have reduced the running time of these detection networks, exposing region proposal computation as a bottleneck. In this work, we introduce a Region Proposal Network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. An RPN is a fully convolutional network that simultaneously predicts object bounds and objectness scores at each position. The RPN is trained end-to-end to generate high-quality region proposals, which are used by Fast R-CNN for detection. We further merge RPN and Fast R-CNN into a single network by sharing their convolutional features—using the recently popular terminology of neural networks with ’attention’ mechanisms, the RPN component tells the unified network where to look. For the very deep VGG-16 model [3] , our detection system has a frame rate of 5 fps ( including all steps ) on a GPU, while achieving state-of-the-art object detection accuracy on PASCAL VOC 2007, 2012, and MS COCO datasets with only 300 proposals per image. In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the foundations of the 1st-place winning entries in several tracks. Code has been made publicly available. <s> BIB004 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> C. OBJECT DETECTION <s> The highest accuracy object detectors to date are based on a two-stage approach popularized by R-CNN, where a classifier is applied to a sparse set of candidate object locations. In contrast, one-stage detectors that are applied over a regular, dense sampling of possible object locations have the potential to be faster and simpler, but have trailed the accuracy of two-stage detectors thus far. In this paper, we investigate why this is the case. We discover that the extreme foreground-background class imbalance encountered during training of dense detectors is the central cause. We propose to address this class imbalance by reshaping the standard cross entropy loss such that it down-weights the loss assigned to well-classified examples. Our novel Focal Loss focuses training on a sparse set of hard examples and prevents the vast number of easy negatives from overwhelming the detector during training. To evaluate the effectiveness of our loss, we design and train a simple dense detector we call RetinaNet. Our results show that when trained with the focal loss, RetinaNet is able to match the speed of previous one-stage detectors while surpassing the accuracy of all existing state-of-the-art two-stage detectors. Code is at: https://github.com/facebookresearch/Detectron . <s> BIB005
Object detection is one of the critical problems to be solved for the development of a complete scene understanding system, which has been recently successfully addressed with deep CNN giving a significant breakthrough BIB002 , BIB003 . In particular, recent advances are mainly driven by the success of region proposal methods and region-based CNN (R-CNN). Furthermore, the R-CNN can train CNN end-toend to classify (SVM classifier) the proposal regions into object categories or background, in which detection accuracy depends on the performance of the region proposal module. Further, Fast R-CNN, which is the improvement version of R-CNN, using ultra-deep networks with a softmax classifier to classify the proposal regions. Generally, the selective search is one of the most popular methods for region proposal module and consumes in much running time as the detection network. Region proposal networks (RPN BIB004 ) is designed to predict region proposals efficiently and to improve region proposal quality. By unifying RPN with Fast R-CNN, the obtained Faster R-CNN can significantly reduce the running time of previous detection networks. Besides, In ILSVRC and COCO 2015 competitions, Faster R-CNN and RPN are the basis of several 1st-place entries in the tracks of ImageNet detection. Further, Mask R-CNN, which based on Faster R-CNN, can predict binary segmentation mask on each region of interest, was proposed to fix the pixel-to-pixel misalignment between network inputs and outputs for Faster R-CNN. Based on ResNet-101, Mask R-CNN [55] surpasses the winner of the 2016 COCO key-point competition, and achieve an average mask precision of 35.7 and running at 5 fps on the COCO test sets. As is known to all, the two-stage object detection method represented by the R-CNN series has performed well in terms of accuracy. This kind of network model conducts object detection in two steps. Firstly, all candidate object regions are selected, and then classification and regression are conducted for each candidate region. However, the main disadvantage of those two-stage methods is its slow detection speed. Different from the idea of two-stage target detection, the onestage method is to divide the region directly on the original image and carry out classification and regression prediction. However, the main disadvantage of the one-stage method is that the positive and negative samples of the bounding box (bbox) are extremely unbalanced, resulting in poor accuracy. To effectively control the ratio of positive and negative samples and prevent the occurrence of imbalance, researchers proposed a new classification loss function Focal loss based on the cross-entropy loss function. To further verify the validity of Focal loss, researchers further designed RetinaNet BIB005 . RetinaNet is a single, unified network composed of a backbone network and two task-specific subnetworks, The backbone is responsible for computing a convolutional feature map over an entire input image and is an off-the-self convolutional network. The first subnet performs convolutional object classification on the backbone's output; the second subnet performs convolutional bounding box regression. The structure of the RetinaNet network is very concise. Note that the original intention of researchers is not to innovate the network structure, but to verify the effectiveness of Focal Loss. Experimental results show that RetinaNet can achieve an excellent balance in recognition accuracy and speed. In TABLE 5, We summarize some benchmark datasets and corresponding deep learning methods commonly used in object detection. Compared with the classical models, these new models are also intended to highlight novel framework/architecture design and relatively excellent generalization performance. In addition to the target detection methods described above, the H-ELM-based fast detection algorithm consists of two parts BIB001 ; one is that a sliding window is used to extract a fixed-size image patch, the other is to design classifier based on H-ELM. It is worth pointing out that this framework can achieve excellent performance on some simple object detection datasets without any additional samples preprocessing. For practical and complex computer vision applications, the ELM series still need to explore the functions of robust feature extraction and classifier.