One of the successful approaches in semi-supervised learning is based on the consistency loss between different predictions under random perturbations. Typically, a student model is trained to be consistent with teachers prediction for the inputs under different perturbation. However, to be successful,the teachers pseudo labels must have good quality, otherwise the whole learning process will fail. Unfortunately, existing methods do not assess the quality of teachers pseudo labels. In this paper, we propose a novel certainty-driven consistency loss (CCL) that exploits the predictive uncertainty information in the consistency loss to let the student dynamically learn from reliable targets. Specifically, we propose two approaches, i.e. Filtering CCL and Temperature CCL to either filter out uncertain predictions or pay less attention on the uncertain ones in the consistency regularization. We combine the two approaches, which we call FT-CCL to further improve consistency learning framework. Based on our experiments, FT-CCL shows improvements on a general semi-supervised learning task and robustness to noisy labels. We further introduce a novel mutual learning method, where one student is decoupled from its teacher, and learns from the other student's teacher, in order to learn additional knowledge. Experimental results demonstrate the advantages of our method over the state-of-the-art semi-supervised deep learning methods.
@cite_4 apply the concept of temperature in model distillation, which aims to distill the knowledge from a large pre-trained network to a much smaller network without lossing much of the generalization ability. The temperature, a hyperparameter inside softmax function, is used to soften the probability distributions of softmax, which encourages the small model to learn more "dark knowledge" distributions from the large model, rather than the hard label. However, the method needs to set the value of temperature empirically, which is shared by all training samples. Our method can automatically define the temperature of each training sample according to its uncertainty, and use its own temperature to decide how much influence it has on training the student model.
@cite_4
2811423591 2951168573 2620998106 2952193981
The parameterization of moist convection contributes to uncertainty in climate modeling and numerical weather prediction. Machine learning (ML) can be used to learn new parameterizations directly from high-resolution model output, but it remains poorly understood how such parameterizations behave when fully coupled in a general circulation model (GCM) and whether they are useful for simulations of climate change or extreme events. Here, we focus on these issues using idealized tests in which an ML-based parameterization is trained on output from a conventional parameterization and its performance is assessed in simulations with a GCM. We use an ensemble of decision trees (random forest) as the ML algorithm, and this has the advantage that it automatically ensures conservation of energy and non-negativity of surface precipitation. The GCM with the ML convective parameterization runs stably and accurately captures important climate statistics including precipitation extremes without the need for special training on extremes. Climate change between a control climate and a warm climate is not captured if the ML parameterization is only trained on the control climate, but it is captured if the training includes samples from both climates. Remarkably, climate change is also captured when training only on the warm climate, and this is because the extratropics of the warm climate provides training samples for the tropics of the control climate. In addition to being potentially useful for the simulation of climate, we show that ML parameterizations can be interrogated to provide diagnostics of the interaction between convection and the large-scale environment. Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, that is better suited to low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy where, rather than one way transfer between a static pre-defined teacher and a student, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on CIFAR-100 recognition and Market-1501 person re-identification benchmarks. Surprisingly, it is revealed that no prior powerful teacher network is necessary -- mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher. Model distillation is an effective and widely used technique to transfer knowledge from a teacher to a student network. The typical application is to transfer from a powerful large network or ensemble to a small network, in order to meet the low-memory or fast execution requirements. In this paper, we present a deep mutual learning (DML) strategy. Different from the one-way transfer between a static pre-defined teacher and a student in model distillation, with DML, an ensemble of students learn collaboratively and teach each other throughout the training process. Our experiments show that a variety of network architectures benefit from mutual learning and achieve compelling results on both category and instance recognition tasks. Surprisingly, it is revealed that no prior powerful teacher network is necessary - mutual learning of a collection of simple student networks works, and moreover outperforms distillation from a more powerful yet static teacher. We introduce a Deep Boltzmann Machine model suitable for modeling and extracting latent semantic representations from a large unstructured collection of documents. We overcome the apparent difficulty of training a DBM with judicious parameter tying. This parameter tying enables an efficient pretraining algorithm and a state initialization scheme that aids inference. The model can be trained just as efficiently as a standard Restricted Boltzmann Machine. Our experiments show that the model assigns better log probability to unseen data than the Replicated Softmax model. Features extracted from our model outperform LDA, Replicated Softmax, and DocNADE models on document retrieval and document classification tasks.