A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves ‘knowledge’ from multiple deep nets, referred to as teachers, to a new deep net model, called the student. The structure of the teachers and the student can differ arbitrarily and they can be trained on entirely different tasks with different output spaces too. Upon training with knowledge flow the student is independent of the teachers. We demonstrate our approach on a variety of supervised and reinforcement learning tasks, outperforming fine-tuning and other ‘knowledge exchange’ methods.

 Research communities have amassed a sizable number of deep net architectures for different tasks, and new ones are added almost daily. Some of those architectures are trained from scratch while others are fine-tuned, i.e., before training, their weights are initialized using a structurally similar deep net which was trained on different data.Beyond fine-tuning, particularly in reinforcement learning, teachers have also been considered in one way or another by BID23 ; BID6 ; BID30 ; BID13 ; BID0 BID21 ; BID2 ; BID26 ; BID20 . For instance, progressive neural net BID23 keeps multiple teachers during both training and inference, and learns to extract useful features from the teachers for a new target task. PathNet BID6 uses genetic algorithms to choose pathways from a giant network for learning new tasks. 'Growing a Brain' BID30 fine-tunes a neural network while growing the network's capacity (wider or deeper layers). Actor-mimic BID20 pre-trains a big model on multiple source tasks, then the big model is used as a weight initialization for a new model which will be trained on a new target task. Knowledge distillation BID9 distills knowledge from a large ensemble of models to a smaller student model. However, all the aforementioned techniques have limitations. For example, progressive neural net models BID23 grow with the number of teachers. This large number of parameters limits the number of teachers a progressive neural net can handle, and largely increases the training and testing time. In PathNet BID6 , searching over a big network for pathways is computationally intensive. For fine-tuning based methods such as 'Growing a Brain' BID30 and actor-mimic BID20 , only one pretrained model can be used at a time. Hence, their performance heavily relies on the chosen pretrained model.To address these shortcomings, we develop knowledge flow which moves 'knowledge' of multiple teachers when training a student. Irrespective of how many teachers we use, the student is guaranteed to become independent at the final stage of training and the size of the resulting student net remains constant. In addition, our framework makes no restrictions on the deep net size of the teacher and student, which provides flexibility in choosing teacher models. Importantly, our approach is applicable to a variety of tasks from reinforcement learning to fully-supervised training.We evaluate knowledge flow on a variety of tasks from reinforcement learning to fully-supervised learning. In particular, we follow BID23 ; BID6 and compare on the same ∞ k=0 γ k r t+k , where γ is the discount factor. The expected future reward when observing state x and when following policy π θπ is defined as V π θπ (x t ) = E τ ∼π θπ [R t |x t ], where τ = {(x t , a t , r t ), (x t+1 , a t+1 , r t+1 ), . . .} is a trajectory generated by following π θπ from state x t .The goal of reinforcement learning is to find a policy that maximizes the expected future reward from each state x t . Without loss of generality, in this paper, we follow the asynchronous advantage actor-critic (A3C) formulation BID17 . In A3C, the policy mapping π θπ (x) = arg max a∈Aπθπ (a|x) is obtained from a probability distribution over states, wherê π θπ (a|x) is modeled by a deep net with parameters θ π . The value function is also approximated by a deep net V θv (x), having parameters θ v .To optimize the policy parameters θ π given a state x t , a loss function based on a scaled negative log-likelihood and a negative entropy regularizer is common: DISPLAYFORM0 [− logπ θπ (a t |x t )(R t − V θv (x t )) − βH(π θπ (·|x t ))] .Hereby, R t = k−1 i=0 γ i r t+i + γ k V θv (x t+k ) is the empirical k-step return obtained when starting in state x t , and |τ | is the length of the trajectory τ generated by following π θπ . The scalar β ≥ 0 is a user-specified constant, and H(π θπ (·|x t )) is the entropy function, which encourages exploration by favoring a uniform probability distributionπ θπ (a|x). To optimize the value function V θv , it is common to use the squared loss DISPLAYFORM1 By minimizing the empirical expectation of τ π (θ π ) and τ v (θ v ), i.e., by addressing DISPLAYFORM2 alternatingly, we learn a policy and a value function that maximize expected return. We developed a general knowledge flow approach that permits to train a deep net from any number of teachers. We showed results for reinforcement learning and supervised learning, demonstrating improvements compared to training from scratch and to fine-tuning. In the future we plan to learn when to use which teacher and how to actively swap teachers during training of a student. BID9 to distill knowledge from a larger model (teacher) to a smaller model (student). The student models have 50% -5% parameters of the teacher models. Following their setup, we conduct experiments on MNIST, MNIST with digit '3' missing in the training set, CIFAR-100, and ImageNet. For MNIST and MNIST with digit '3' missing, following KD, the teacher model is an MLP with two hidden layers of 1200 hidden units, and the student model is an MLP with two hidden layers of 800 hidden units. For CIFAR-100, we use the model from Chen FORMULA2 as teacher model. The student model follows the structure of the teacher, but the number of output channels of each convolutional layer is halved. For ImageNet, the teacher model is a 50-layer ResNet BID8 , and the student model is a 18-layer ResNet. The test error of the distilled student model are summarize in TAB4 . Our framework has consistently better performance than KD, because the student model in our framework benefits not only from the output layer behavior of the teacher but also from intermediate layer representations of the teacher.
‘Knowledge Flow’ trains a deep net (student) by injecting information from multiple nets (teachers). The student is independent upon training and performs very well on learned tasks irrespective of the setting (reinforcement or supervised learning).
{"src": "scitldr-AIC_train", "paper_id": "BJeOioA9Y7"}