Knowledge transfer in machine learning is commonly used to leverage a model trained on a source task to improve training a model for a corresponding target task. One prevalent method is Transfer Learning [1]}, [2]}, [3]}, which allows for reusing knowledge learned on a source model to a target model by recycling learned parameters and limiting further training to the lower layers of the source model. Transfer Learning can improve the time it takes to learn the new task in the same domain as the source task and the final performance of the model. Distilling knowledge from an ensemble of source models [4]}, [5]} is another approach for knowledge transfer where the original set or a subset of the data used to train the source models is leveraged. Distilling knowledge from source to target is done by defining a cross-entropy loss between outputs of source and target softmax layers. Recently, a method for zero-shot knowledge distillation [6]}, [7]} was proposed, where a transfer set is extracted directly from the source model by sampling the Dirichlet distribution learned for each class for softmax probabilities. Sampled probabilities are then used to construct Data Impressions that correspond to model output per class, replacing conventional input data. Transferring knowledge can also be achieved by using neuron activation [8]}, [9]}. Here, instead of extracting knowledge by considering the magnitude of neuron responses to data, minimizing the difference in neuron activation between source and target is used to train our target model. When transferring knowledge between models, some information learned on the source model might be insignificant or even harmful to the target model.
Additionally, exploring where data should be injected into the target model can benefit our knowledge transfer. Therefore, defining meta models to decide what data should be transferred and to what location in the target model could positively impact knowledge transfer [10]}. While knowledge transfer predominantly deals with training a target model once a source model has finished its training process, we are interested in the case where actively learning models can still benefit each other in real-time. Moreover, while knowledge transfer methods are largely evaluated on their ability to learn a comparable but distinct task, in our problem set-up, we wish to accurately inherit the source model's ability to classify or predict the data it was trained on initially.
