Semi-supervised learning (SSL) is a study that efficiently exploits a large amount of unlabeled data to improve performance in conditions of limited labeled data. Most of the conventional SSL methods assume that the classes of unlabeled data are included in the set of classes of labeled data. In addition, these methods do not sort out useless unlabeled samples and use all the unlabeled data for learning, which is not suitable for realistic situations. In this paper, we propose an SSL method called selective self-training (SST), which selectively decides whether to include each unlabeled sample in the training process. It is also designed to be applied to a more real situation where classes of unlabeled data are different from the ones of the labeled data. For the conventional SSL problems which deal with data where both the labeled and unlabeled samples share the same class categories, the proposed method not only performs comparable to other conventional SSL algorithms but also can be combined with other SSL algorithms. While the conventional methods cannot be applied to the new SSL problems where the separated data do not share the classes, our method does not show any performance degradation even if the classes of unlabeled data are different from those of the labeled data. Recently, machine learning has achieved a lot of success in various fields and well-refined datasets are considered to be one of the most important factors (Everingham et al., 2010; Krizhevsky et al., 2012; BID6 . Since we cannot discover the underlying real distribution of data, we need a lot of samples to estimate it correctly (Nasrabadi, 2007) . However, creating a large amount of dataset requires a huge amount of time, cost and manpower BID3 .Semi-supervised learning (SSL) is a method relieving the inefficiencies in data collection and annotation process, which lies between the supervised learning and unsupervised learning in that both labeled and unlabeled data are used in the learning process (Chapelle et al., 2009; BID3 . It can efficiently learn a model from fewer labeled data using a large amount of unlabeled data BID15 . Accordingly, the significance of SSL has been studied extensively in the previous literatures BID18 BID5 Kingma et al., 2014; BID4 BID2 . These results suggest that SSL can be a useful approach in cases where the amount of annotated data is insufficient.However, there is a recent research discussing the limitations of conventional SSL methods BID3 . They have pointed out that conventional SSL algorithms are difficult to be applied to real applications. Especially, the conventional methods assume that all the unlabeled data belong to one of the classes of the training labeled data. Training with unlabeled samples whose class distribution is significantly different from that of the labeled data may degrade the performance of traditional SSL methods. Furthermore, whenever a new set of data is available, they should be trained from the scratch using all the data including out-of-class 1 data.In this paper, we focus on the classification task and propose a deep neural network based approach named as selective self-training (SST) to solve the limitation mentioned above. Unlike the conventional self-training methods in (Chapelle et al., 2009) , our algorithm selectively utilizes the unlabeled data for the training. To enable learning to select unlabeled data, we propose a selection network, which is based on the deep neural network, that decides whether each sample is to be added or not. Different from BID12 , SST does not use the classification results for the data selection. Also, we adopt an ensemble approach which is similar to the co-training method BID0 ) that utilizes outputs of multiple classifiers to iteratively build a new training dataset. In our case, instead of using multiple classifiers , we apply a temporal ensemble method to the selection network. For each unlabeled instance, two consecutive outputs of the selection network are compared to keep our training data clean. In addition, we have found that the balance between the number of samples per class is quite important for the performance of our network. We suggest a simple heuristics to balance the number of selected samples among the classes. By the proposed selection method, reliable samples can be added to the training set and uncertain samples including out-of-class data can be excluded.SST is a self-training framework, which iteratively adopts the newly annotated training data (details in Section 2.1). SST is also suitable for the incremental learning which is frequently used in many real applications when we need to handle gradually incoming data. In addition, the proposed SST is suitable for lifelong learning which makes use of more knowledge from previously acquired knowledge BID10 Carlson et al., 2010; Chen & Liu, 2018) . Since SSL can be learned with labeled and unlabeled data , any algorithm for SSL may seem appropriate for lifelong learning. However, conventional SSL algorithms are inefficient when out-of-class samples are included in the additional data. SST only add samples having high relevance in-class data and is suitable for lifelong learning. The main contributions of the proposed method can be summarized as follows:• For the conventional SSL problems, the proposed SST method not only performs comparable to other conventional SSL algorithms but also can be combined with other algorithms.• For the new SSL problems, the proposed SST does not show any performance degradation even with the out-of-class data.• SST requires few hyper-parameters and can be easily implemented.• SST is more suitable for lifelong learning compared to other SSL algorithms .To prove the effectiveness of our proposed method, first, we conduct experiments comparing the classification errors of SST and several other state-of-the-art SSL methods (Laine & Aila, 2016; BID9 Luo et al., 2017; Miyato et al., 2017) in conventional SSL settings. Second, we propose a new experimental setup to investigate whether our method is more applicable to realworld situations. The experimental setup in BID3 samples classes among in-classes and out-classes . In the experimental setting in this paper, we sample unlabeled instances evenly in all classes. (details in Section 6.6 of the supplementary material). We evaluate the performance of the proposed SST using three public benchmark datasets : CIFAR-10, CIFAR-100 BID8 Hinton, 2009), and SVHN (Netzer et al., 2011) . We proposed selective self-training (SST) for semi-supervised learning (SSL) problem. Unlike conventional methods, SST selectively samples unlabeled data and trains the model with a subset of the dataset. Using selection network, reliable samples can be added to the new training dataset. In this paper, we conduct two types of experiments. First, we experiment with the assumption that unlabeled data are in-class like conventional SSL problems. Then, we experiment how SST performs for out-of-class unlabeled data.For the conventional SSL problems, we achieved competitive results on several datasets and our method could be combined with conventional algorithms to improve performance. The accuracy of SST is either saturated or not depending on the dataset. Nonetheless, SST has shown performance improvements as a number of data increases. In addition, the results of the combined experiments of SST and other algorithms show the possibility of performance improvement.For the new SSL problems, SST did not show any performance degradation even if the model is learned from in-class data and out-of-class unlabeled data. Decreasing the threshold of the selection network in new SSL problem, performance degrades. However, the output of the selection network shows different trends according to in-class and out-of-class. By setting a threshold that does not add out-of-class data, SST has prevented the addition of out-of-class samples to the new training dataset. It means that it is possible to prevent the erroneous data from being added to the unlabeled dataset in a real environment. 6 SUPPLEMENTARY MATERIAL
Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.
{"src": "scitldr-AIC_train", "paper_id": "SyzrLjA5FQ"}