Parameters in deep neural networks which are trained on large-scale databases can generalize across multiple domains, which is referred as “transferability”. Unfortunately, the transferability is usually defined as discrete states and it differs with domains and network architectures. Existing works usually heuristically apply parameter-sharing or fine-tuning, and there is no principled approach to learn a parameter transfer strategy. To address the gap, a Parameter Transfer Unit (PTU) is proposed in this paper. PTU learns a fine-grained nonlinear combination of activations from both the source domain network and the target domain network, and subsumes hand-crafted discrete transfer states. In the PTU, the transferability is controlled by two gates which are artificial neurons and can be learned from data. The PTU is a general and flexible module which can be used in both CNNs and RNNs. It can be also integrated with other transfer learning methods in a plug-and-play manner. Experiments are conducted with various network architectures and multiple transfer domain pairs. Results demonstrate the effectiveness of the PTU as it outperforms heuristic parameter-sharing and fine-tuning in most settings.
The most relevant work to the proposed method is the cross-stitch network @cite_10 . Instead of assigning blocks of parameters as transferable'' or not transferable'', a soft parameter-based transfer method is adopted. Knowledge sharing between networks in two tasks is achieved with a cross-stitch'' unit. It learns a linear combination of activations from different networks. The proposed PTU is different from the cross-stitch network in the following two aspects. First, the transferability is controlled by linear combination coefficients in the cross-stitch unit, while the PTU learns a non-linear combination which is more expressive. Secondly, the cross-stitch unit is proposed for multi-task learning with CNNs while the PTU is designed for transfer learning where the target domain performance is the main focus. Moreover, the PTU is applied and evaluated with both CNNs and RNNs.
@cite_10
2963877604
Multi-task learning in Convolutional Networks has displayed remarkable success in the field of recognition. This success can be largely attributed to learning shared representations from multiple supervisory tasks. However, existing multi-task approaches rely on enumerating multiple network architectures specific to the tasks at hand, that do not generalize. In this paper, we propose a principled approach to learn shared representations in ConvNets using multitask learning. Specifically, we propose a new sharing unit: "cross-stitch" unit. These units combine the activations from multiple networks and can be trained end-to-end. A network with cross-stitch units can learn an optimal combination of shared and task-specific representations. Our proposed method generalizes across multiple tasks and shows dramatically improved performance over baseline methods for categories with few training examples.