Semi-supervised learning, i.e. jointly learning from labeled an unlabeled samples, is an active research topic due to its key role on relaxing human annotation constraints. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different  perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100 and Mini-ImageNet despite being much simpler than other state-of-the-art. These results demonstrate that pseudo-labeling can outperform consistency regularization methods, while the opposite was supposed in previous work. Code will be made available. Convolutional neural networks (CNNs) have become the dominant approach in computer vision (Lin et al., 2017; Liu et al., 2018; Kim et al., 2018; Xie et al., 2018) . To best exploit them, vast amounts of labeled data are required. Obtaining such labels, however, is not trivial, and the research community is exploring alternatives to alleviate this (Li et al., 2017; Oliver et al., 2018; Liu et al., 2019) . Knowledge transfer via deep domain adaptation (Wang & Deng, 2018 ) is a popular alternative that seeks to learn transferable representations from source to target domains by embedding domain adaptation in the learning pipeline. Other approaches focus exclusively on learning useful representations from scratch in a target domain when annotation constraints are relaxed (Oliver et al., 2018; Arazo et al., 2019; Gidaris et al., 2018) . Semi-supervised learning (SSL) (Oliver et al., 2018) focuses on scenarios with sparsely labeled data and extensive amounts of unlabeled data; learning with label noise (Arazo et al., 2019) seeks robust learning when labels are obtained automatically and may not represent the image content; and self-supervised learning (Gidaris et al., 2018) uses data supervision to learn from unlabeled data in a supervised manner. This paper focuses on SSL for image classification, a recently very active research area (Li et al., 2019) . SSL is a transversal task for different domains including images (Oliver et al., 2018) , audio (Zhang et al., 2016) , time series (González et al., 2018) , and text (Miyato et al., 2016) . Recent approaches in image classification primarily focus on exploiting the consistency in the predictions for the same sample under different perturbations (consistency regularization) (Sajjadi et al., 2016; Li et al., 2019) , while other approaches directly generate labels for the unlabeled data to guide the learning process (pseudo-labeling) (Lee, 2013; Iscen et al., 2019) . These two alternatives differ importantly in the mechanism they use to exploit unlabeled samples. Consistency regularization and pseudo-labeling approaches apply different strategies such as a warm-up phase using labeled data (Tarvainen & Valpola, 2017; Iscen et al., 2019) , uncertainty weighting (Shi et al., 2018; Li et al., 2019) , adversarial attacks (Miyato et al., 2018; Qiao et al., 2018) , or graph-consistency (Luo et al., 2018; Iscen et al., 2019) . These strategies deal with confirmation bias (Tarvainen & Valpola, 2017; Li et al., 2019) , also known as noise accumulation (Zhang et al., 2016) . This bias stems from using incorrect predictions on unlabeled data for training in subsequent epochs and, thereby increasing confidence in incorrect predictions and producing a model that will tend to resist new changes. This paper explores pseudo-labeling for semi-supervised deep learning from the network predictions and shows that, contrary to previous attempts on pseudo-labeling (Iscen et al., 2019; Oliver et al., 2018; Shi et al., 2018) , simple modifications to prevent confirmation bias lead to state-of-the-art performance without adding consistency regularization strategies. We adapt the approach proposed by Tanaka et al. (2018) in the context of label noise and apply it exclusively on unlabeled samples. Experiments show that this naive pseudo-labeling is limited by confirmation bias as prediction errors are fit by the network. To deal with this issue, we propose to use mixup augmentation as an effective regularization that helps calibrate deep neural networks (Thulasidasan et al., 2019) and, therefore, alleviates confirmation bias. We find that mixup alone does not guarantee robustness against confirmation bias when reducing the amount of labeled samples or using certain network architectures (see Subsection 4.3), and show that, when properly introduced, dropout regularization (Srivastava et al., 2014) and data augmentation mitigates this issue. Our purely pseudo-labeling approach achieves state-of-the-art results (see Subsection 4.4) without requiring multiple networks (Tarvainen & Valpola, 2017; Qiao et al., 2018; Li et al., 2019; Verma et al., 2019) , nor does it require over a thousand epochs of training to achieve peak performance in every dataset (Athiwaratkun et al., 2019; Berthelot et al., 2019) , nor needs many (ten) forward passes for each sample (Li et al., 2019) . Compared to other pseudo-labeling approaches, the proposed approach is simpler in that it does not require graph construction and diffusion (Iscen et al., 2019) or combination with consistency regularization methods (Shi et al., 2018) , but still achieves state-of-the-art results. This paper presented a semi-supervised learning approach for image classification based on pseudolabeling. We proposed to directly use the network predictions as soft pseudo-labels for unlabeled data together with mixup augmentation, a minimum number of labeled samples per mini-batch, dropout and data augmentation to alleviate confirmation bias. This conceptually simple approach outperforms related work in four datasets, demonstrating that pseudo-labeling is a suitable alternative to the dominant approach in recent literature: consistency-regularization. The proposed approach is, to the best of our knowledge, both simpler and more accurate than most recent approaches. Future work should explore SSL in class-unbalanced and large-scale datasets, synergies of pseudo-labelling and consistency regularization, and careful hyperparameter tuning. Figure 3 presents the cross-entropy loss for labeled samples when training with 13-CNN, WR-28 and PR-18 and using 500 and 250 labels in CIFAR-10. This loss is a good indicator of a robust convergence to reasonable performance as the interquartile range for cases failing (250 labels for WR-28 and PR-18) is much higher.
Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.
{"src": "scitldr-AIC_train", "paper_id": "rJel41BtDH"}