Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability. We demonstrate that 1) Transferred models tend to find flatter minima, since their weight matrices stay close to the original flat region of pretrained parameters when transferred to a similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitzness, which accelerates and stabilizes training substantially. The improvement largely attributes to the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in back-propagation. 3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages in that the transferability first increases during training, and then declines. We further provide a theoretical analysis to verify our observations. The last decade has witnessed the enormous success of deep neural networks in a wide range of applications. Deep learning has made unprecedented advances in many research fields, including computer vision, natural language processing, and robotics. Such great achievement largely attributes to several desirable properties of deep neural networks. One of the most prominent properties is the transferability of deep feature representations. Transferability is basically the desirable phenomenon that deep feature representations learned from one dataset can benefit optimization and generalization on different datasets or even different tasks, e.g. from real images to synthesized images, and from image recognition to object detection (Yosinski et al., 2014) . This is essentially different from traditional learning techniques and is often regarded as one of the parallels between deep neural networks and human learning mechanisms. In real-world applications, practitioners harness transferability to overcome various difficulties. Deep networks pretrained on large datasets are in prevalent use as general-purpose feature extractors for downstream tasks (Donahue et al., 2014) . For small datasets, a standard practice is to fine-tune a model transferred from large-scale dataset such as ImageNet (Russakovsky et al., 2015) to avoid over-fitting. For complicated tasks such as object detection, semantic segmentation and landmark localization, ImageNet pretrained networks accelerate training process substantially (Oquab et al., 2014; He et al., 2018) . In the NLP field, advances in unsupervised pretrained representations have enabled remarkable improvement in downstream tasks (Vaswani et al., 2017; Devlin et al., 2019) . Despite its practical success, few efforts have been devoted to uncovering the underlying mechanism of transferability. Intuitively, deep neural networks are capable of preserving the knowledge learned on one dataset after training on another similar dataset (Yosinski et al., 2014; Li et al., 2018b; 2019) . This is even true for notably different datasets or apparently different tasks. Another line of works have observed several detailed phenomena in the transfer learning of deep networks (Kirkpatrick et al., 2016; Kornblith et al., 2019 ), yet it remains unclear why and how the transferred representations are beneficial to the generalization and optimization perspectives of deep networks. The present study addresses this important problem from several new perspectives. We first probe into how pretrained knowledge benefits generalization. Results indicate that models fine-tuned on target datasets similar to the pretrained dataset tend to stay close to the transferred parameters. In this sense, transferring from a similar dataset makes fine-tuned parameters stay in the flat region around the pretrained parameters, leading to flatter minima than training from scratch. Another key to transferability is that transferred features make the optimization landscape significantly improved with better Lipschitzness, which eases optimization. Results show that the landscapes with transferred features are smoother and more predictable, fundamentally stabilizing and accelerating training especially at the early stages of training. This is further enhanced by the proper scaling of gradient in back-propagation. The principal component of gradient is suppressed in the transferred weight matrices, controlling the magnitude of gradient and smoothing the loss landscapes. We also investigate a common concern raised by practitioners: when is transfer learning helpful to target tasks? We test the transferability of pretrained networks with varying inputs and labels. Instead of the similarity between pretrained and target inputs, what really matters is the similarity between the pretrained and target tasks, i.e. both inputs and labels are required to be sufficiently similar. We also investigate the relationship between pretraining epoch and transferability. Surprisingly, although accuracy on the pretrained dataset increases throughout training, transferability first increases at the beginning and then decreases significantly as pretraining proceeds. Finally, this paper gives a theoretical analysis based on two-layer fully connected networks. Theoretical results consistently justify our empirical discoveries. The analysis here also casts light on deeper networks. We believe the mechanism of transferability is the fundamental property of deep neural networks and the in-depth understanding presented here may stimulate further algorithmic advances. Why are deep representations pretrained from modern neural networks generally transferable to novel tasks? When is transfer learning feasible enough to consistently improve the target task performance? These are the key questions in the way of understanding modern neural networks and applying them to a variety of real tasks. This paper performs the first in-depth analysis of the transferability of deep representations from both empirical and theoretical perspectives. The results reveal that pretrained representations will improve both generalization and optimization performance of a target network provided that the pretrained and target datasets are sufficiently similar in both input and labels. With this paper, we show that transfer learning, as an initialization technique of neural networks, exerts implicit regularization to restrict the networks from escaping the flat region of pretrained landscape.
Understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability.
{"src": "scitldr-AIC_train", "paper_id": "BylKL1SKvr"}