Finally, we perform standard knowledge distillation on the finetuned model using the unlabeled target data to improve the model by taking advantage of the soft pseudo-labels. We use the same architecture for both student and teacher networks. This is similar to [1]}, [2]} except that we do distillation on unlabeled target data rather than labeled data. Similar to [1]}, [2]}, our intuition is that this method will reduce the generalization gap caused by using one-hot encoding of the ground truth in the first stage.
