Another approach is Knowledge Distillation, also known as Teacher-Student model. It was originally developed as a model compression method [1]}. The Teacher network first learns the representations and outputs prediction labels. Then the Student network is trained on the prediction labels of Teacher. One key aspect of Knowledge Distillation is that it can transfer knowledge across models with different structure. It even enables human-machine transfer learning: [2]} implemented this type of learning, albeit indirectly. The researchers first trained a Teacher model on an FER dataset and obtained a saliency map for each image by visualizing the activations of the Teacher model. Then the researchers masked the images by leaving only the most important parts of the image based on the saliency maps - usually around the eyes and mouth - and used the masked images to train the Student model. The masked images initially helped accelerate the training, but the acceleration was retained only if the training data was switched to unmasked images after a certain point - there was no effect on accuracy, however. The critical aspect was that the researchers validated the masked images by comparing them to eye-tracking results by human observers. As the saliency maps of Teacher model were shown to be similar to human attention maps, this work is an indirect example of human-machine transfer learning.
