One of the successful approaches in semi-supervised learning is based on the consistency loss between different predictions under random perturbations. Typically, a student model is trained to be consistent with teachers prediction for the inputs under different perturbation. However, to be successful,the teachers pseudo labels must have good quality, otherwise the whole learning process will fail. Unfortunately, existing methods do not assess the quality of teachers pseudo labels. In this paper, we propose a novel certainty-driven consistency loss (CCL) that exploits the predictive uncertainty information in the consistency loss to let the student dynamically learn from reliable targets. Specifically, we propose two approaches, i.e. Filtering CCL and Temperature CCL to either filter out uncertain predictions or pay less attention on the uncertain ones in the consistency regularization. We combine the two approaches, which we call FT-CCL to further improve consistency learning framework. Based on our experiments, FT-CCL shows improvements on a general semi-supervised learning task and robustness to noisy labels. We further introduce a novel mutual learning method, where one student is decoupled from its teacher, and learns from the other student's teacher, in order to learn additional knowledge. Experimental results demonstrate the advantages of our method over the state-of-the-art semi-supervised deep learning methods.
A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then to average their predictions. Unfortunately, making predictions using a whole ensemble of models is cumbersome and may be too computationally expensive to allow deployment to a large number of users, especially if the individual models are large neural nets. Caruana and his collaborators have shown that it is possible to compress the knowledge in an ensemble into a single model which is much easier to deploy and we develop this approach further using a different compression technique. We achieve some surprising results on MNIST and we show that we can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model. We also introduce a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. Unlike a mixture of experts, these specialist models can be trained rapidly and in parallel.