The ever-increasing size of modern datasets combined with the difficulty of obtaining label information has made semi-supervised learning of significant practical importance in modern machine learning applications. In comparison to supervised learning, the key difficulty in semi-supervised learning is how to make full use of the unlabeled data. In order to utilize manifold information provided by unlabeled data, we propose a novel regularization called the tangent-normal adversarial regularization, which is composed by two parts. The two parts complement with each other and jointly enforce the smoothness along two different directions that are crucial for semi-supervised learning. One is applied along the tangent space of the data manifold, aiming to enforce local invariance of the classifier on the manifold, while the other is performed on the normal space orthogonal to the tangent space, intending to impose robustness on the classifier against the noise causing the observed data deviating from the underlying data manifold.   Both of the two regularizers are achieved by the strategy of virtual adversarial training. Our method has achieved state-of-the-art performance on semi-supervised learning tasks on both artificial dataset and practical datasets. The recent success of supervised learning (SL) models, like deep convolutional neural networks, highly relies on the huge amount of labeled data. However, though obtaining data itself might be relatively effortless in various circumstances, to acquire the annotated labels is still costly, limiting the further applications of SL methods in practical problems. Semi-supervised learning (SSL) models, which requires only a small part of data to be labeled, does not suffer from such restrictions. The advantage that SSL depends less on well-annotated datasets makes it of crucial practical importance and draws lots of research interests. The common setting in SSL is that we have access to a relatively small amount of labeled data and much larger amount of unlabeled data. And we need to train a classifier utilizing those data. Comparing to SL, the main challenge of SSL is how to make full use of the huge amount of unlabeled data, i.e., how to utilize the marginalized input distribution p(x) to improve the prediction model i.e., the conditional distribution of supervised target p(y|x). To solve this problem, there are mainly three streams of research.The first approach, based on probabilistic models, recognizes the SSL problem as a specialized missing data imputation task for classification problem. The common scheme of this method is to establish a hidden variable model capturing the relationship between the input and label, and then applies Bayesian inference techniques to optimize the model BID10 Zhu et al., 2003; BID21 . Suffering from the estimation of posterior being either inaccurate or computationally inefficient, this approach performs less well especially in high-dimensional dataset BID10 .The second line tries to construct proper regularization using the unlabeled data, to impose the desired smoothness on the classifier. One kind of useful regularization is achieved by adversarial training BID8 , or virtual adversarial training (VAT) when applied to unlabeled data BID15 . Such regularization leads to robustness of classifier to adversarial examples, thus inducing smoothness of classifier in input space where the observed data is presented. The input space being high dimensional, though, the data itself is concentrated on a underlying manifold of much lower dimensionality BID2 BID17 Chapelle et al., 2009; BID22 . Thus directly performing VAT in input space might overly regularize and does potential harm to the classifier. Another kind of regularization called manifold regularization aims to encourage invariance of classifier on manifold BID25 BID0 BID18 BID11 BID22 , rather than in input space as VAT has done. Such manifold regularization is implemented by tangent propagation BID25 BID11 or manifold Laplacian norm BID0 BID13 , requiring evaluating the Jacobian of classifier (with respect to manifold representation of data) and thus being highly computationally inefficient.The third way is related to generative adversarial network (GAN) BID7 . Most GAN based approaches modify the discriminator to include a classifier, by splitting the real class of original discriminator into K subclasses, where K denotes the number of classes of labeled data BID24 BID19 BID5 BID20 . The features extracted for distinguishing the example being real or fake, which can be viewed as a kind of coarse label, have implicit benefits for supervised classification task. Besides that, there are also works jointly training a classifier, a discriminator and a generator BID14 .Our work mainly follows the second line. We firstly sort out three important assumptions that motivate our idea:The manifold assumption The observed data presented in high dimensional space is with high probability concentrated in the vicinity of some underlying manifold of much lower dimensionality BID2 BID17 Chapelle et al., 2009; BID22 . We denote the underlying manifold as M. We further assume that the classification task concerned relies and only relies on M BID22 . The noisy observation assumption The observed data x can be decomposed into two parts as x = x 0 + n, where x 0 is exactly supported on the underlying manifold M and n is some noise independent of x 0 BID1 BID21 . With the assumption that the classifier only depends on the underlying manifold M, the noise part might have undesired influences on the learning of the classifier. The semi-supervised learning assumption If two points x 1 , x 2 ∈ M are close in manifold distance, then the conditional probability p(y|x 1 ) and p(y|x 2 ) are similar BID0 BID22 BID18 . In other words, the true classifier, or the true condition distribution p(y|X) varies smoothly along the underlying manifold M. DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 c 0 w R T 9 e + O m m f W V l n i K i e 7 2 8 f e R P y f 1 y 8 p 3 R 7 U U h c l o R Y P g 9 J S A e U w S R G G 0 q A g V T n C h Z F u V x A j b r g g l 7 X v Q g g f n / y U H G 9 0 w 6 A b f t 1 s 7 3 2 a x b H A V t l 7 t s Z C t s X 2 2 G d 2 x H p M s B / s i t 2 w W + / S u / b u v P u H 0 j l v 1 v O O / Q P v 9 x 8 6 h a h y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f j E y Y 9 J 5 C G f U l R v p 4 t P 7 5 Z Z a A t s = " > A DISPLAYFORM4 c 0 w R T 9 e + O m m f W V l n i K i e 7 2 8 f e R P y f 1 y 8 p 3 R 7 U U h c l o R Y P g 9 J S A e U w S R G G 0 q A g V T n C h Z F u V x A j b r g g l 7 X v Q g g f n / y U H G 9 0 w 6 A b f t 1 s 7 3 2 a x b H A V t l 7 t s Z C t s X 2 2 G d 2 x H p M s B / s i t 2 w W + / S u / b u v P u H 0 j l v 1 v O O / Q P v 9 x 8 6 h a h y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f j E y Y 9 J 5 C G f U l R v p 4 t P 7 5 Z Z a A t s = " > A DISPLAYFORM5 c 0 w R T 9 e + O m m f W V l n i K i e 7 2 8 f e R P y f 1 y 8 p 3 R 7 U U h c l o R Y P g 9 J S A e U w S R G G 0 q A g V T n C h Z F u V x A j b r g g l 7 X v Q g g f n / y U H G 9 0 w 6 A b f t 1 s 7 3 2 a x b H A V t l 7 t s Z C t s X 2 2 G d 2 x H p M s B / s i t 2 w W + / S u / b u v P u H 0 j l v 1 v O O / Q P v 9 x 8 6 h a h y < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f j E y Y 9 J 5 C G f U l R v p 4 t P 7 5 Z Z a A t s = " > A DISPLAYFORM6 c 0 w R T 9 e + O m m f W V l n i K i e 7 2 8 f e R P y f 1 y 8 p 3 R 7 U U h c l o R Y P g 9 J S A e U w S R G G 0 q A g V T n C h Z F u V x A j b r g g l 7 X v Q g g f n / y U H G 9 0 w 6 A b f t 1 s 7 3 2 a x b H A V t l 7 t s Z C t s X 2 2 G d 2 x H p M s B / s i t 2 w W + / S u / b u v P u H 0 j l v 1 v O O / Q P v 9 x 8 6 h a h y < / l a t e x i t > x 0 < l a t e x i t s h a 1 _ b a s e 6 4 = " U v o A x a M + o b w r T 4 7 t H V I T M R l 7 n k U = " > A A A C P n i c b V B N a 9 t A E F 2 l T e o q X 2 5 y 7 DISPLAYFORM7 4 2 c d e G 4 N 8 / + S E Y n n Z 9 r + t / f t 3 q n 6 3 j a J C X 5 B X p E J + 8 I X 3 y g Z y T A e H k K / l O f p J f z r X z w / n t / L m 1 b j j r m U N y p 5 y / / w C 0 + a o E < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v o A x a M + o b w r T 4 7 t H V I T M R l 7 n k U = " > A A A C P n i c b V B N a 9 t A E F 2 l T e o q X 2 5 y 7 DISPLAYFORM8 4 2 c d e G 4 N 8 / + S E Y n n Z 9 r + t / f t 3 q n 6 3 j a J C X 5 B X p E J + 8 I X 3 y g Z y T A e H k K / l O f p J f z r X z w / n t / L m 1 b j j r m U N y p 5 y / / w C 0 + a o E < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " U v o A x a M + o b w r T 4 7 t H V I T M R l 7 n k U = " > A A A C P n i c b V B N a 9 t A E F 2 l T e o q X 2 5 y 7 DISPLAYFORM9 We present the tangent-normal adversarial regularization, a novel regularization strategy for semisupervised learning, composing of regularization on the tangent and normal space separately. The tangent adversarial regularization enforces manifold invariance of the classifier, while the normal adversarial regularization imposes robustness of the classifier against the noise contained in the observed data. Experiments on artificial dataset and multiple practical datasets demonstrate that our approach outperforms other state-of-the-art methods for semi-supervised learning. The performance of our method relies on the quality of the estimation of the underlying manifold, hence the breakthroughs on modeling data manifold could also benefit our strategy for semi-supervised learning, which we leave as future work. represent two different classes. The observed data is sampled as x = x 0 + n, where x 0 is uniformly sampled from M and n ∼ N (0, 2 −2 ). We sample 6 labeled training data, 3 for each class, and 3, 000 unlabeled training data, as shown in FIG9 .
We propose a novel manifold regularization strategy based on adversarial training, which can significantly improve the performance of semi-supervised learning.
{"src": "scitldr-AIC_train", "paper_id": "BJxz5jRcFm"}