A Survey on the New Generation of Deep Learning in Image Processing <s> 4) RESNET <s> Deeper neural networks are more difficult to train. We present a residual learning framework to ease the training of networks that are substantially deeper than those used previously. We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions. We provide comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth. On the ImageNet dataset we evaluate residual nets with a depth of up to 152 layers—8× deeper than VGG nets [40] but still having lower complexity. An ensemble of these residual nets achieves 3.57% error on the ImageNet test set. This result won the 1st place on the ILSVRC 2015 classification task. We also present analysis on CIFAR-10 with 100 and 1000 layers. The depth of representations is of central importance for many visual recognition tasks. Solely due to our extremely deep representations, we obtain a 28% relative improvement on the COCO object detection dataset. Deep residual nets are foundations of our submissions to ILSVRC & COCO 2015 competitions1, where we also won the 1st places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation. <s> BIB001 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 4) RESNET <s> Deep neural networks, albeit their great success on feature learning in various computer vision tasks, are usually considered as impractical for online visual tracking, because they require very long training time and a large number of training samples. In this paper, we present an efficient and very robust tracking algorithm using a single convolutional neural network (CNN) for learning effective feature representations of the target object in a purely online manner. Our contributions are multifold. First, we introduce a novel truncated structural loss function that maintains as many training samples as possible and reduces the risk of tracking error accumulation. Second, we enhance the ordinary stochastic gradient descent approach in CNN training with a robust sample selection mechanism. The sampling mechanism randomly generates positive and negative samples from different temporal distributions, which are generated by taking the temporal relations and label noise into account. Finally, a lazy yet effective updating scheme is designed for CNN training. Equipped with this novel updating algorithm, the CNN model is robust to some long-existing difficulties in visual tracking, such as occlusion or incorrect detections, without loss of the effective adaption for significant appearance changes. In the experiment, our CNN tracker outperforms all compared state-of-the-art methods on two recently proposed benchmarks, which in total involve over 60 video sequences. The remarkable performance improvement over the existing trackers illustrates the superiority of the feature representations, which are learned purely online via the proposed deep learning framework. <s> BIB002 </s> A Survey on the New Generation of Deep Learning in Image Processing <s> 4) RESNET <s> This paper introduces the sparse regularization for the convolutional neural network (CNN) with the rectified linear units (ReLU) in the hidden layers. By introducing the sparseness for the inputs of the ReLU, there is effect to push the inputs of the ReLU to zero in the learning process. Thus it is expected that the unnecessary increase of the outputs of the ReLU can be prevented. This is the similar effect with the Batch Normalization. Also the unnecessary negative values of the inputs of the ReLU can be reduced by introducing the sparseness. This can improve the generalization of the trained network. The relations between the proposed approach and the Batch Normalization or the modifications of the activation function such as Exponential Linear Unit (ELU) are also discussed. The effectiveness of the proposed method was confirmed through the detail experiments. <s> BIB003
It is well-known that many other non-trivial visual recognition tasks have also greatly benefited from ultra-deep models. Therefore, a question arises that '' is learning better networks as easy as stacking more layers?'' With the network layer increasing, it is indisputable that an obstacle is the notorious problem of vanishing or exploding gradients, which can hamper the convergence of networks. Besides, accuracy gets saturated and then degrades rapidly. Unexpectedly, such degradation does not cause by over-fitting. Furthermore, adding more layers to a suitably deep model lead to higher training error. For the degradation problem, The classical ResNet BIB001 can explicitly let these stacked layers fit a residual mapping rather than a desired underlying mapping. For example, a building block in Fig.4 that has two layers, Therefore, a building block defined as: where x and y are the input and output of the layers considered. The function F(x) represents the residual mapping to be learned, that is, where σ denotes classical active function ReLU BIB003 . The operation F(x) + x is performed by a short-cut connection and element-wise addition. It should note that the dimensions of x and F(x) must be the same. If this is not the case, the first choice is that one can perform a linear projection W s by the short-cut connections to match the dimensions: The other choice is that one can still perform identity mapping, but with extra zero entries padded for increasing dimensions was considered. Certainly, we can also use the square matrix in Eq. BIB002 , and it can further make the form of residual function F(x) more flexible. In particular, if the residual function F(x) represents multiple convolutional layers, then the element-wise addition is performed on two feature maps, channel by channel. More importantly, ResNet structure is simple, which can solves the problem of deep convolution neural network performance degradation under ultra-deep conditions, and its classification performance is excellent. Finally, more tricks include BN right after each convolution and before activation, and one can do not use dropout or max-out.