Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP. The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks. In many real-world applications, deep learning practitioners often have limited number of training instances. Direct training a deep neural network with a small training data set usually results in the so-called over-fitting problem and the quality of the obtained model is low. A simple yet effective approach to obtain high-quality deep learning models is to perform weight fine-tuning. In such practices, a deep neural network is first trained using a large (and possibly irrelevant) source dataset (e.g. ImageNet). The weights of such a network are then fine-tuned using the data from the target application domain.Fine-tuning is a specific approach to perform transfer learning in deep learning. The weights pretrained by the source dataset with a sufficiently large number of instances usually provide a better initialization for the target task than random initializations. In a typical fine-tuning approach, weights in lower convolution layers are fixed and weights in upper layers are re-trained using data from the target domain. In this approach parameters of the target model may be driven far away from initial values, which also causes over-fitting in transfer learning scenarios.Approaches called regularization using the starting point as the reference (SPAR), were recently proposed to solve the over-fitting problem. For example, Li et al. BID10 proposed L 2 -SP that incorporates the Euclid distance between the target weights and the starting point (i.e., weights of source network) as part of the loss. Minimizing this loss function, L 2 -SP aims to minimize the empirical loss of deep learning while reducing the distance of weights between source and target networks. They achieved significant improvement compared with standard practice of using the weight decay (L 2 normalization).However such regularization method may not deliver optimal solution for transfer learning. On one side, if the regularization is not strong, even with fine-turning, the weights may still be driven far away from the initial position, leading to the lose of useful knowledge, i.e. catastrophic memory loss. On the other side, if the regularization is too strong, newly obtained model is constrained to a local neighborhood of the original model, which may be suboptimal to the target data set. Although aforementioned methods demonstrated the power of regularization in deep transfer learning, we argue that we need to perform research on at least the following two aspects in order to further improve current regularization methods.Behavior vs. Mechanisms. The practice of weight regularization for CNN is motivated by a simple intuition -the network (layers) with similar weights should produce similar outputs. However, due to the complex structures of deep neural network with strong redundancies, regulating the model parameters directly seems an over-killing of the problem. We argue that we should regularize the "Behavior", or in our case, the outer layer outputs (e.g. the feature maps) produced by each layer, rather than model parameters. With constrained feature maps, the generalization capacity could be improved through aligning the behaviors of the outer layers of the target network to the source one, which has been pre-trained using an extremely large dataset. In Convolutional Neural Networks, which we focus on exclusively in this paper, an outer layer is a convolution layer and the output of an outer layer is its feature map.Syntax vs Semantics. While regularizing the feature maps might improve the transfer of generalization capacity, it is still difficult to design such regularizers. It is challenging to measure the similarity/distance between the feature maps without understanding its semantics or representations. For example for image classification, some of the convolution kernels may be corresponding to features that are shared between the two learning tasks and hence should be preserved in transfer learning while others are specific to the source task and hence could be eliminated in transfer learning.In this paper, we propose a novel regularization approach DELTA to address the two issues. Specifically, DELTA selects the discriminative features from outer layer outputs through re-weighting the feature maps with a novel supervised attention mechanism. Through paying attention to discriminative parts of feature maps, DELTA characterizes the distance between source/target networks using their outer layer outputs, and incorporates such distance as the regularization term of the loss function. With the back-propagation, such regularization finally affects the optimization for weights of deep neural network and awards the target network generalization capacity inherited from the source network.In summary, our key insight is what we call "unactivated channel re-usage". Specifically our approach identifies those transferable channels and preserves such filters through regularization and identify those untransferable channels and reuse them, using an attention mechanism with feature map regularization.We have conducted extensive experiments using a wide range of source/target datasets and compared DELTA to the existing deep transfer learning algorithms that are in pursuit of weight similarity. The experiment results show that DELTA significantly outperformed the state-of-the-art regularization algorithms including L 2 and L 2 -SP with higher accuracy on a wide group of image classification data sets.The rest of the paper is organized as follows: in Section 2 related works are summarized, in Section 3 our feature map based regularization method is introduced, in Section 4 experimental results are presented and discussed, and finally in Section 5 the paper is concluded. To better understand the performance gain of DELTA we performed an experiment where we analyzed how parameters of the convolution filters change after fine-tuning. Towards that purpose we randomly sampled images from the testing set of Stanford Dogs 120. For ResNet-101, which we use exclusively in this paper, we grouped filters into stages as described in (he et al., 2016) . These stages are conv2 x, conv3 x, conv4 x, conv5 x. Each stage contains a few stacked blocks and a block is a basic inception unit having 3 conv2d layers. One conv2d layer consists of a number of output filters. We flatten each filter into a one dimension parameter vector for convenience. The Euclidian distance between the parameter vectors before and after fine-tuning is calculated. All distances are sorted as shown in FIG3 .We observed a sharp difference between the two distance distributions. Our hypothesis of possible cause of the difference is that simply using L 2 -SP regularization all convolution filters are forced to be similar to the original ones. Using attention, we allow "unactivated" convolution filters to be reused for better image classification. About 90% parameter vectors of DELTA have larger distance than L 2 -SP . We also observe that a small number of filters is driven very far away from the initial value (as shown at the left end of the curves in FIG3 . We call such an effect as "unactivated channel re-usage".To further understand the effect of attention and the implication of "unactivated channel re-usage", we "attributed" the attention to the original image to identify the set of pixels having high contributions in the activated feature maps. We select some convolution filters on which the source model (the initialization before fine-tuning) has low activation. For the convenience of analyzing the effect of regularization methods, each element a i of the original activation map is normalized with DISPLAYFORM0 where the min and max terms in the formula represent for the minimum and maximum value of the whole activation map respectively. Activation maps of these convolution filter for various regularization method are presented on each row.As shown in FIG4 , our first observation is that without attention, the activation maps from DELTA in different images are more or less the same activation maps from other regularization methods. This partially explains the fact that we do not observe significant improvement of DELTA without attention.Using attention, however, changes the activation map significantly. Regularization of DELTA with attention show obviously improved concentration. With attention (the right-most column in FIG4 ), we observed a large set of pixels that have high activation at important regions around the head of the animals. We believe this phenomenon provides additional evidence to support our intuition of "unactivated channel re-usage" as discussed in previous paragraphs. In addition, we included new statistical results of activations on part locations of CUB-200-2011 supporting the above qualitative cases. The CUB-200-2011 datasets defined 15 discriminative parts of birds, e.g. the forehead, tail, beak and so on. Each part is annotated with a pixel location representing for its center position if it is visible. So for each image, we got several key points which are very important to discriminate its category. Using all testing examples of CUB-200-2011, we calculated normalized activations on these key points of these different regularization methods. As shown in TAB2 , DELTA got the highest average activations on those key points, demonstrating that DELTA focused on more discriminate features for bird recognition. In this paper, we studied a regularization technique that transfers the behaviors and semantics of the source network to the target one through constraining the difference between the feature maps generated by the convolution layers of source/target networks with attentions. Specifically, we designed a regularized learning algorithm DELTA that models the difference of feature maps with attentions between networks, where the attention models are obtained through supervised learning. Moreover, we further accelerate the optimization for regularization using start point as reference (SPAR). Our extensive experiments evaluated DELTA using several real-world datasets based on commonly used convolutional neural networks. The experiment results show that DELTA is able to significantly outperform the state-of-the-art transfer learning methods.
improving deep transfer learning with regularization using attention based feature maps
{"src": "scitldr-AIC_train", "paper_id": "rkgbwsAcYm"}