Aside from representation learning, auxiliary pretext-tasks can also help the model generalize better. In [1]}, it is shown that incorporating self-supervised losses during pre-training can improve model robustness. In [2]}, rotation prediction is shown to effectively replace labeled data in conditional GANs to make them completely unsupervised. In [3]}, self-supervision is combined with existing few-shot learning methods to leverage the unlabeled data and boost the model performance. In [4]}, self-supervision is applied to semi-supervised learning by incorporating a supervised loss on a small amount of labeled data while solving the pretext task on the entire dataset. Our simple method is similar to [4]} except that we are applying it to the task of domain adaptation and improve it with self-distillation.
