When training a neural network for a desired task, one may prefer to adapt a pretrained network rather than start with a randomly initialized one -- due to lacking enough training data, performing lifelong learning where the system has to learn a new task while being previously trained for other tasks, or wishing to encode priors in the network via preset weights. The most commonly employed approaches for network adaptation are fine-tuning and using the pre-trained network as a fixed feature extractor, among others. 

 In this paper we propose a straightforward alternative: Side-Tuning. Side-tuning adapts a pretrained network by training a lightweight "side" network that is fused with the (unchanged) pre-rained network using a simple additive process. This simple method works as well as or better than existing solutions while it resolves some of the basic issues with fine-tuning, fixed features, and several other common baselines. In particular, side-tuning is less prone to overfitting when little training data is available, yields better results than using a fixed feature extractor, and doesn't suffer from catastrophic forgetting in lifelong learning.   We demonstrate the performance of side-tuning under a diverse set of scenarios, including lifelong learning (iCIFAR, Taskonomy), reinforcement learning, imitation learning (visual navigation in Habitat), NLP question-answering (SQuAD v2), and single-task transfer learning (Taskonomy), with consistently promising results. The goal of side-tuning is to capitalize on a pretrained model to better learn one or more novel tasks. By design, side-tuning does so without degrading performance of the base model. The framework is straightforward: it assumes access to the frozen base model B : X → Y that maps inputs into some representation space that is shared between the base task and the current (target) task. This representation space is flexible and could either be a latent space (e.g. in R N ) or actual model predictions. Side-tuning then learns a side model S : X → Y, so that the representations for the target task are R(x) B(x) ⊕ S(x), fine-tuning adapts too easily and forgets old information. Side-tuning is a simple method to address these limitations. for some combining operation ⊕. We use a learned alpha-blending, a ⊕ b αa + (1 − α)b for this purpose (other options are discussed in Section 3.0.3). Certain pre-set curricula of α reduce the side-tuning framework to: fine-tuning, feature extration, and stage-wise training (see Fig. 3 , right). Hence those can be viewed as special cases of the general side-tuning framework. Also, other curricula suggest (e.g.) a maximum a posteriori estimator that integrates the B(x) prior with the evidence from S(x). Side-tuning is an example of an additive learning approach as it adds (strategically placed) parameters for each new task. Fixed feature extraction would be a simple example of an additive approach with zero new parameters. As a result, fixed features are don't adapt the base network over the lifetime of the agent. A number of existing approaches address this by learning new parameters (the number of which scales with the size of the base network) for each new task (e.g. . Unlike these approaches, side-tuning places no constraints on the structure of the side network, allowing the parameters to be strategically allocated. In particular, side-tuning can use tiny networks when the base requires only minor updates. By adding fewer parameters per task, side-tuning can learn more tasks before the model grows large enough to require parameter consolidation. These approaches stand in contrast to most existing methods for incremental learning, which do not increase the number of parameters over time and instead gradually fill up the capacity of a large base model. For example, fine-tuning updates all the parameters. A large body of constraint-based methods focus on how to regularize these updates in order to prevent inter-task interference (Cheung et al., 2019) . Side-tuning does not require such regularization since the additive structure means inter-task interference is not possible. We compare side-tuning to alternative approaches on both the iCIFAR and Taskonomy datasets. iCIFAR consists of ten distinct 10-class image classification problems. Taskonomy covers multiple tasks of varied complexity from across computer vision (surface normal and depth estimation, edge detection, image 1000-way classification, etc.). On these datasets, side-tuning uses side networks that are much smaller than the base. Consequently, even without consolidation, side-tuning uses fewer learnable parameters than the alternative methods. This remarkably simple approach deals with the key challenges of incremental learning. Namely, it does not suffer from either: • Catastrophic forgetting: which is the tendency of a network to abruptly lose previously learned knowledge upon learning new information. We show this in Section 4.2.1. • Rigidity: where networks become increasingly unable to adapt to new problems as they accrue constraints from previous problems. We explore this in Section 4.2.2. Side-tuning avoids these problems while remaining highly performant, which we demonstrate in Section 4.2.3. We have introduced the side-tuning framework, a simple yet effective approach for additive learning. Since it does not suffer from catastrophic forgetting or rigidity, it is naturally suited to incremental learning. The theoretical advantages are reflected in empirical results, and side-tuning outperforms existing approaches in challenging contexts and with state-of-the-art neural networks. We further demonstrated that the approach is effective in multiple domains and with various network types.
Side-tuning adapts a pre-trained network by training a lightweight "side" network that is fused with the (unchanged) pre-trained network using a simple additive process.
{"src": "scitldr-AIC_train", "paper_id": "HyxakgrFvS"}