Manifold Adversarial Learning

Recently proposed adversarial training methods show the robustness to both adversarial and original examples and achieve state-of-the-art results in supervised and semi-supervised learning. All the existing adversarial training methods consider only how the worst perturbed examples (i.e., adversarial examples) could affect the model output. Despite their success, we argue that such setting may be in lack of generalization, since the output space (or label space) is apparently less informative.In this paper, we propose a novel method, called Manifold Adversarial Training (MAT). MAT manages to build an adversarial framework based on how the worst perturbation could affect the distributional manifold rather than the output space. Particularly, a latent data space with the Gaussian Mixture Model (GMM) will be first derived.On one hand, MAT tries to perturb the input samples in the way that would rough the distributional manifold the worst. On the other hand, the deep learning model is trained trying to promote in the latent space the manifold smoothness, measured by the variation of Gaussian mixtures (given the local perturbation around the data point). Importantly, since the latent space is more informative than the output space, the proposed MAT can learn better a robust and compact data representation, leading to further performance improvement. The proposed MAT is important in that it can be considered as a superset of one recently-proposed discriminative feature learning approach called center loss. We conducted a series of experiments in both supervised and semi-supervised learning on three benchmark data sets, showing that the proposed MAT can achieve remarkable performance, much better than those of the state-of-the-art adversarial approaches. We also present a series of visualization which could generate further understanding or explanation on adversarial examples.


I. INTRODUCTION
A DVERSARIAL examples refer to augmented data points generated by imperceptible perturbation of input samples. Being difficult to distinguish from real examples, such adversarial examples could however change the prediction of many of the best learning models including the state-ofthe-art deep learning models [1] [2] [3]. To alleviate such problems, researchers have proposed adversarial training, able to certify both the robustness on adversarial examples and the generalization on original examples. Adversarial training could be used in both supervised and semi-supervised training. In supervised adversarial learning, the data labels are needed to derive the worst perturbation against loss function [4] [5]; in semi-supervised adversarial learning, a virtual adversarial training (VAT) is used by smoothing the output distribution with penalizing the KL-divergence between outputs of adversarial and original examples. VAT achieves state-of the-art performance on both image and text classification [6] [7] [8].
Previous adversarial training methods simply consider how to make the results of prediction worse (in the output space) without considering how the data are robustly represented in a latent space. In general, the latent space is much more informative than the output space. It is hence meaningful if we can design the adversarial learning in the latent space rather than the output space. In this work, we develop a novel model called Manifold Adversarial Training (MAT) in the latent space. We engage an information based regularization, i.e., Maximum Mutual Information (MMI) [9] [10] so as to define a distributional manifold in the latent space. We then apply the adversarial training to smooth such manifold by penalizing the KL-divergence between the distributions of latent features of the adversarial and original examples. The novel framework is trained in an adversarial way: the adversarial noise is generated to rough the distributional manifold, while the model is trained to smooth it to make the latent space more representative. It is similar to traditional Laplacian regularization methods with locality-preserving properties [11] [12]. However, our approach is based on information geometry and the information metric KL-divergence.
To our best knowledge, this is one novel work that learns adversarially both a robust and compact representation in the latent space. It also presents a unified framework in that a simplified MAT could derive a famous recently-proposed discriminative feature learning model [13] [14]. Though developed in the framework of supervised learning, it is straightforward and much easier to be extended in semi-supervised learning. We develop a feasible and efficient training algorithm capable of obtaining remarkably better performance than the existing adversarial training approaches. In particular, we implemented our proposed method on benchmark datasets MNIST, CIFAR-10, and SVHN. Our method achieves in supervised and semi-supervise learning the state-of-the-art performance, much better than the best of the existing counterpart methods.

II. RELATED WORK
It has a long history to use the perturbed examples to regularize the output [15]. Bishop et al. proposed a method to add the Gaussian noise to input samples and showed that it is equivalent to adding the penalty term to original objective function [16]. Dropout can also be treated as random perturbation to prevent from over fitting [15] [17]. The Unified Gradient Regularization Family is proposed to find the worst perturbation to increase the objective function [5]. It approximates the non-convex problem with Taylor series and applies the Lagrange multiplier method to evaluate the worst perturbation. Another similar work proposed by Aman et al. is to perturb the underlying data distribution in a Wasserstein ball [4]. Virtual Adversarial training proposed by Takeru [6] [8] is perhaps most related to our work. It developed a method extending the adversarial training to semi-supervised task by promoting the local smoothness of the output distribution.
Another type of semi-supervised learning methods are based on generative models. Ladder network combined the deep network and auto encoder with connections between two networks at each layer and achieves encouraging results [18]. Triple generative Adversarial Network is proposed to combine Generative Adversarial Network (GAN) with classifier. There are three players, generator, discriminator and classifier playing against with each other [19]. Some Bayesian methods employ variational methods with deep learning [20].
This work is also related to some traditional manifold regularization. Laplacian Eigenmaps was developed to consider how to construct a representation for data lying on a low dimensional manifold embedded in a high-dimensional space [11]. A geometric framework was also proposed to exploit the geometry of the marginal distribution using the information of both labeled and unlabeled examples [12]. Both of these two methods tried to smooth the low dimensional manifold with preserving local neighborhood information.

III. MAIN METHOD
In this section, we first introduce two previous adversarial training methods, adversarial training with l p norm constraint and Virtual Adversarial Training (VAT) which are close related to our proposed approach. Then we describe our proposed Manifold Adversarial Training (MAT). Specifically, we first introduce how to regularize the latent space with Maximum Mutual Information (MMI) and represent the latent features with Gaussian Mixtures. We then define the notion of the low dimensional submanifold of latent space and the smoothness of statistic manifold. After that we describe the major framework and present a series of theory to derive the practical optimization algorithm. Finally, we also briefly conduct a computational analysis. We now give a set of notations. The input set is represented as . Let x i ∈ R I and y i ∈ R O where, I is the dimension of input space and O denotes the output dimension. The model distribution is denoted by p m (y|x, θ) with model parameters θ. In this paper, labeled dataset where q(y|x l ) is the true distribution of output label given input and p m (.) denotes the model distribution. ǫ is a perturbation added on the input within a small range. σ is a small value. To implement the adversarial training, the inner maximization problem need to be firstly solved to obtain the worst perturbation. For simplifying optimization, the first Taylor expansion of D(.) is used as the approximation for original objective function. With the l p constraint, the worst perturbation can be approximated as: where p * is the dual of p, i.e, 1 p * + 1 p = 1 and ∇L the first derivative of loss function with respect to input x. When p = ∞, this method can be degraded to Fast Gradient Sign Method (FGSM). And the worst perturbation becomes: For finding the worst perturbation, it is easy to compute ∇L with backpropagation. Moreover it does not need a large amount of computation since it just needs one more backward process. After computing the adversarial perturbation, the model distribution p m (y|x, θ) is trained to approximate the true distribution q(y|x) on both natural and adversarial examples. It has been proved to be able to achieve the better generalization performance than the traditional training methods of deep neural network.

B. Virtual Adversarial Training
Another work similar to our method is Virtual Adversarial Training (VAT). In most cases, the full labeled information is not provided and it is important to make use of the information of unlabeled data. Different from the traditional adversarial training methods, Virtual Adversarial Training method can utilize both labeled and unlabeled data. The optimization problem of VAT can be formulated as: where, x is either labeled data or unlabeled data. Different from traditional adversarial training, Virtual Adversarial Training minimize the divergence between two model distributions instead of the divergence between true distribution and model distribution. The objective of VAT is to smooth the output distribution around x through the method of adversarial training. Specifically, inner optimization problem is to find the worst perturbation ǫ within a small range to make the model distributions of x and x + ǫ the most different, oppositely, the outer one is to try to make two model distributions the same. Therefore, VAT does not require the label information and can be applied on unsupervised and semi-supervised tasks.

C. Manifold Adversarial Training
Both the traditional adversarial training and Virtual Adversarial Training methods attempt to find the worst perturbation to make the outputs of models worst. However, these two methods do not consider how the data is represented in latent space. Therefore, we design a new adversarial method Manifold Adversarial Training (MAT) to learn a good geometric structure of the data distribution for improving the robustness of representation in latent space. In following subsections, we first introduce how to regularize the latent space with Maximum Mutual Information (MMI). We then define the notion of smoothness of distributional manifold. After that we describe the major framework and present a series of theory to derive the practical optimization algorithm. Finally, we also briefly conduct a computational analysis.

D. Modeling the Latent Space
In the probability theory, the mutual information is to measure the mutual dependence between two variables. In other words, it measures how much knowing one of these variables reduces uncertainty about the other. For learning discriminative features with more information of labels, MMI is proposed to increase the mutual dependence between the latent features and the class label which can be seen as an information regularization. We define the label set {c j } M j=1 and the latent features where M and N denote the number of classes and samples respectively. D is the dimensionality of the latent space. The mutual information between the event C = c j and the event Z = z i with respect to θ is given by: where C and Z are two random variables representing class label and latent feature respectively. In this work, we assume the conditional probability P θ (z i |c j ), the multivariate Gaussian distribution, as Eq. (2). We also assume that the latent features are generated by Gaussian mixtures. The number of components of Gaussian mixtures is given as the class number in this paper.
In Eq.
(2), µ j is the class center for class c j and Σ j is corresponding covariance matrix. D represents the dimension of latent space. In this work, we assume the class prior P (c j ) is a constant 1/M (M is the class number). Then, the latent feature can be represented by Gaussian mixtures, Again,the number of the components is set to the class number M . Each component of Gaussian mixtures denotes the probability of the latent feature z i assigned to class c j . We reformulate the mutual information I θ (c j , z i ) as follows: In this case, maximizing the mutual information is equivalent to maximizing the log posterior logP (c j |z i ). The last term of Eq. (7) is the log marginal distribution which can be seen as the normalization term. Therefore, we can just maximize the first term logN (z i |µ j , Σ j ). It is equivalent to making the latent features more compact or discriminate with respect to class centers. Then we define the information regularization in this paper: One of our objectives is to maximize the above information regularization term so as to model a good latent space. In this paper, though we assume GMM over the latent space, it presents a generalized version to a recently proposed famous center loss model [13]. This can be readily obtained in Proposition 0.1.
Proposition III.1. The center loss [13] can be viewed as the special case of our proposed information regularization.
Proof. We reformulate the information regularization as: If we make the covariance matrix as identity, we can have: where C is a constant. It is clear that the above last term changes into the center loss as defined in [13].
Remarks. Center loss [13] is also proposed to penalize latent features to be closer to class centers. However, it implicitly assumes an identity covariance matrix as shown in Proposition 0.1. In comparison, we propose a more generalized regularization. With such regularization, we can represent the latent features with Gaussian mixtures which can be easily employed on information geometry and information metric KL-divergence, which we show in the next subsection. Note that, in this work, we just employ such regularization term in the last hidden layer.

E. Defining Low Dimensional Statistical Manifold
An n-dimensional manifold M is defined as a set of points such that each point has n-dimensional extensions in its neighborhood and such a neighborhood is topologically equivalent to an n-dimensional Euclidean space. Intuitively speaking, it can be viewed as a deformed Euclidean space.
In this paper, the dimension of the entire manifold S 0 of latent space is denoted as D and the coordinate system is defined as ζ = (ζ 1 , ζ 2 , ..., ζ D ). In the previous section, the feature points are modeled by Gaussian mixtures. Hence, the set of all the Gaussian mixtures is an M -dimensional statistical submanifold S 1 , where a point on manifold S 1 denotes a Gaussian mixture function and u = (u 1 , u 2 , ..., u M ) describes a coordinate system (u i is the i th component of Gaussian mixture) as illustrated in Figure 1. The degree of separation of two points on manifold S 1 is measured by the KL-divergence between two Gaussian mixtures. In this paper, our aim is to smooth the manifold S 1 . Fig. 1. The region of circle represents the entire manifold S 0 of latent space defined by the coordinate system ζ = (ζ 1 , ζ 2 , ..., ζ D ) and the grey region in circle denotes a submanifold S 1 defined by the coordinate system u = (u 1 , u 2 , ..., u M ). The degree of separation between two points p 1 and p 2 is described by KL-divergence.

F. Defining Statistical Manifold Smoothing
In this section, we consider how to evaluate the smoothness of statistical manifold. First, we define some notations. The input set is denoted as {x i } N i=1 and the output of model is defined as The training set is given as follows: We then use D to train the model distribution P θ (y|x, θ) with the regularization term I θ . In the previous subsection, the latent features are represented by the Gaussian mixtures u(z i ). We can then easily define the distance between the two points on the statistical manifold with KL-divergence KL[p||q] between mixtures p and q. For learning a good manifold in the latent space, we try to preserve the local information in the embedding as in [11] [12]. Specifically, we add the small adversarial perturbation to input samples and try to make close the latent representation of perturbed samples and original one. Since the latent features are represented by Gaussian mixtures in this work, the information metric KL-divergence can be readily used to measure the similarity. The notion of smoothness of statistical manifold can then be defined as the variation of the latent features on the statistical space caused by the adversarial perturbation ǫ as follows: where f (x, θ) denotes the latent feature of x with the model parameters θ. P GM (f (x)) = 1 M M j=1 N (f (x)|µ j , Σ j ) which is defined as the Gaussian mixture representation for latent features f (x). ǫ represents the adversarial perturbation.
The smaller value of M KL (x, θ) means the statistical manifold is more smooth at x even when the perturbation is imposed. We also define the smoothness in the output or label space as in [6] [8]: In more details, L KL (x, θ) describes the KL-divergence between the output space of adversarial example and the original examples. The overall smoothness for both the output and manifold in latent space are then given as follows:

G. Final Optimization Problem of MAT
In the previous subsections, we have defined the information regularization, manifold smoothness, and label space smoothness. We can now obtain the final optimization problem of our proposed framework as follows: where the hyperparameters λ 1 > 0, λ 2 > 0 and σ > 0.
In the above objective function, the first term represents the data log likelihood, the second term defines the overall data smoothness in both the manifold and output space, and the last term presents the information regularization. On one hand, the proposed MAT framework tries to optimize the model parameter θ so as to find the best latent space (by maximizing the information regularization I θ ), enlarge the data log likelihood (so as to fit the data) as well as increasing the data smoothness (decreasing N i=1 logS kl (x i , θ)); on the other hand, the imperceptible perturbation ǫ tries to minimize the above objective function. In other words, the proposed novel model tries to find the best parameter θ that is even robust to the worst perturbation as given by the optimal ǫ. Since this adversarial learning is defined in the latent manifold space, we call this model as Manifold Adversarial Training (MAT). In the next section, we will use a strategy similar to [5] and [6] and discuss how to solve this optimization problem practically and efficiently.

H. Practical Algorithm
Similar to [5], we try to solve the above optimization problem in an alternative way. We first solve the inner minimization problem with respect to ǫ, i.e. the worst perturbation, which we denote as ǫ adv . In order to calculate this worst perturbation, the KL-divergence between GMMs needs to be calculated firstly. Since it is difficult to conduct the evaluation directly, we approximate it by matching between the Gaussian elements of the two Gaussian mixture density as described in [21]: where Ψ ij is a K × K stochastic matrix. For simplifying the calculation, we try to optimize the upper bound of KLdivergence with assuming Ψ ij an identity matrix and the mixture weight α = 1/K and β = 1/K. Since it is difficult to evaluate the worst perturbation in a non-convex problem, we relax it to a convex problem with second-order Taylor expansions as in [6] [8]. Since M KL (ǫ, x, θ) reaches the minimum value when ǫ = 0 and P (y|x, θ) is differentiable with respect to x and θ, the first derivative ∇ ǫ M KL (ǫ, x, θ) = 0. We can finally approximate it as follows: where, H(x, θ) is the Hessian matrix calculated by employing the second derivative, H(x, θ) ≡ ∇∇ ǫ M KL (ǫ, x, θ)| ǫ=0 . Before we could obtain the worst perturbation, we first present Lemma III.2 and Lemma III.3 as follows: Lemma III.2. Let B be a real square matrix in R × R and u be the dominant eigenvector of B and e is a vector which is not perpendicular to u. Then, the iterative calculation of e ← Be (15) will make e converge to u. ( . represents the normalization operator) Lemma III.3. Let A be a Hessian matrix function F (x) with respect to x in R × R and r be a vector in R. Then we have: Proof. We give the Taylor expansion of the first derivative of function F (x + ar): then we have: By using Lemma 0.2 and Lemma 0.3 , we could solve the inner minimization problem, i.e., obtain the current worst perturbation. Specifically, we have Theorem 0.4 showing that it can then be written as the production of most sensitive directiond and the scale ǫ, where the most sensitive direction d can be approximated iteratively by the power method. (19) where, . represents the normalization operator and ξ > 0. d is a vector which is not perpendicular to dominant eigenvector of H(x, θ).
Theorem III.4. The steepest directiond in (14) can be approximated by iterative calculation of: where . represents the normalization operator.
Proof. Applying the Lagrange multiplier method on (14), we can easily get that the steepest directiond has the same direction with the dominant eigenvector of H(x, θ). Then we can easily approximate the steepest directiond by iteritive calculation of (refer to Lemma 0.2): where, d is a vector which is not perpendicular to dominant eigenvector of H(x, θ). Then we can calculate Bv using Lemma 0.3: where, ξ is a small value. Then we have: After we solve the inner minimization problem with respect to ǫ, we can then solve the maximization problem with respect to θ. We iterate these two steps until the process converges. The detailed pseudo code algorithm is shown in Algorithm 1.

I. Computational Analysis
We briefly touch the computational analysis here. Compared with previous methods like VAT [6], our proposed method needs to compute the mean and covariance matrix for Gaussian mixtures additionally. In this work, parameters of Gaussian mixtures are evaluated by the latent features of labeled data instead of stochastic gradient. It may be inefficient to compute the inverse of covariance matrix. However, in the lowdimensional latent space, the inverse of covariance matrix can be easily and directly obtained; even in the very highdimensional space, the covariance matrix can be assumed as a diagonal matrix which is also easy to get its inverse. Same as VAT, we just need one iteration of the power method to estimate the worst perturbation ǫ adv . The whole procedure for updating the parameters of deep neural networks consists of Update the parameters of neural network with stochastic gradient: Update the covariance matrix and mean of Gaussian Mixtures using the latent features: 11: for i in 1...m do 12: end for 15: end for two forward and two back propagations. The first forward and backward propagations are used to evaluate the worst perturbation. After calculating the final loss, propagate forward and backward again to update the parameters of neural network and Gaussian mixtures.

IV. EXPERIMENT
For assessing the efficacy of our proposed MAT, we implemented it on several benchmark datasets including MNIST, CIFAR-10, and SVHN. In principle, adversarial methods can be regarded as a robust method which typically have better generalization abilities than the traditional methods. To check if the proposed MAT can indeed improve the classification performance, we first applied MAT on the on benign data of MNIST and CIFAR-10 in both the supervised and semisupervised task. Additionally, to check further the robustness of our proposed adversarial training framework, we perform various experiments to examine how the proposed MAT could defend the attack from various adversarial methods. To illustrate our proposed method, a series of visualizations were made, offering some interesting results which might be used to explain the adversarial examples.

A. Experiments on Benign Data
We first evaluated the performance of our proposed MAT methods against many other comparison methods on the benign data. We first report the performance of various methods in the setting of supervised classification and then perform the comparison in semi-supervised learning.
1) Supervised Learning: We implement the same framework Lenet++ with [13] on MNIST dataset. For Lenet++, there are only two dimensions in the last hidden layer which is convenient for visualization. The based framework for experiment on CIFAR-10 is the same as [6] called Conv-Large. For MNIST dataset, we train the deep model with 60, 000 labeled training samples, and we evaluate it with 10, 000 test samples. For CIFAR-10, we use 50, 000 training samples and 10, 000 test samples. To search good hyper parameters, the training set is divided into 50, 000 training set and 10, 000 validation set. We choose a set of hyper parameters with the best performance on the validation set. For MNIST dataset, the best hyper parameters are obtained as: λ 1 = 0.1, λ 2 = 0.1 and σ = 2. For CIFAR-10, the parameters are obtained as: λ 1 = 0.01, λ 2 = 0.1 and σ = 20.  Table I and Table II list the performance of our proposed method and other competitive methods on MNIST and CIFAR-10. Generally, adversarial methods can be regarded as a robust method which typically have better generalization abilities than the traditional methods. As observed from both the tables, MAT clearly achieves the best performance among all the comparison methods in both the datasets. It is probably unfair to compare with some methods, e.g., ladder network, simply  II  TEST PERFORMANCE ON CIFAR-10 IN SUPERVISED LEARNING   Method  CIFAR-10 Test error rate(%) Network in Network [22] 8.81 All-CNN [23] 7.25 Deeply Supervised Net [24] 7.97 Highway Network [25] 7.72 RPT [6] 6.25 ± 0.04 Baseline 6.76 ± 0.07 VAT 5.81 ± 0.02 MAT 4.40 ± 0.03 because the different based model were used there. However, it is sufficient to show the superiority of our proposed MAT due to its significant lower error rate than most of the other competitive adversarial methods.
2) Semi-supervised Learning: In the methodology part, we have introduced the MAT's final objective function Eq. (13) where the first term is the ordinary soft-max loss function and the last term is the information regularization term. These two terms are exploited on the training data with labels. And the second term is used to smooth the manifold and output distribution which does not need the label information. Therefore, our method can readily be extended to semi-supervised learning. Following the same setting as [6], we implement our MAT both on CIFAR-10 and SVHN with 4, 000 labeled data and 1, 000 labeled data respectively. We use the same base model with [6] called Cov-Large with batch normalization and dropout. We exploit the mini-batch of size 32 for both labeled data and unlabeled data on CIFAR-10. For SVHN, we use the labeled batch with size 32 and unlabeled batch with size 128.

RPT [6]
18.56 ± 0.29 VAT 14.82 ± 0.13 MAT 12.85 ± 0.21 Table III and Table IV show the performance of our proposed model as well as recent state-of-the-art semi-supervised learning methods on CIFAR-10 and SVHN. Once again, our proposed method MAT achieves the highest performance, which further shows its advantages.
3) Visualization: In order to illustrate why the proposed MAT could perform excellent, we take the MNIST as one example to visualize various methods. In particular, Figure 2 shows the illustration of the last embedding space of Lenet++ with center loss, softmax, VAT, and MAT on the MNIST test data. 1 It is obvious that the latent features of our method are represented more compactly and discriminatively with respect to class centers. VAT learns the similar latent space with softmax, but apparently both of their features are not as discriminative as our proposed MAT. This may illustrate why our proposed MAT method could usually generate better classification performance than VAT and the traditional CNN.
Since our proposed MAT seeks a smooth latent space which we believe would benefit the classification, we plot in Figure 3 the learning curve (accuracy rate) and the smoothness for the three different methods: MAT (our proposed method), VAT, and the baseline (traditional CNN) on CIFAR-10. 2 For MAT, the best set of hyperparameters, i.e., λ 1 = 0.01 λ 2 = 0.1 and σ = 20 was used, while for VAT, the best setting reported in [6] was directly applied. Compared with the other two methods, our proposed MAT clearly increases the smoothness R smooth , which is defined by the average of the smoothness of output space and latent space: When we calculate the smoothness, the perturbation needs to be normalized to unit vector. Figure 3(b) shows that our proposed method indeed learns the smoother latent and output space.

B. Performance on Defending Adversarial Examples
We now turn to examining how the proposed MAT method could defend the attacks from various adversarial examples generation approaches in comparison with other competitive methods. Visualization is also presented so as to obtain further understandings on adversarial examples.
1) Robustness to Adversarial Attacks: We implement different methods on FGSM and 2-norm adversarial attack for MNIST, CIFAR-10, and SVHN datasets. Particularly, we generate in the test sets of MNIST and CIFAR-10 10,000 adversarial examples according to FSGM and 2-norm attacks [5] respectively. For SVHN, we generate 26,032 adversarial examples. We increase the level of adversarial noise gradually from 0 to 8 in MNIST and SVHN with the step size as 1 and from 0 to 13 in CIFAR-10 with the step size as 1.6. We then test the performance of various training methods on these adversarial examples. The performance is plotted in Fig. 4. As clearly observed, the proposed MAT shows better robustness against the two types of adversarial examples. Particularly, when the adversarial noises are small, all the adversarial training methods show similar results but perform much better than the CNN (exploiting no adversarial training); when the adversarial attacks are stronger, the proposed method overall demonstrates clearly better performance in almost all the cases, verifying its significant robustness. One exception can be identified in Fig. 4(c) where the FGSM training method performs the best while our proposed MAT performs the second. However, since the FGSM training was specifically designed to defend the FGSM adversarial examples, it will not be very surprising that FGSM can do better against the FGSM adversarial examples. Nonetheless, except Fig. 4(c), our MAT still demonstrates the best robustness in all the other cases.
Again, taking MNIST as one example, we show the TSNE embedding for the second last layer given by VAT and MAT under various levels of 2-norm adversarial attack. As Figure 5 shows, when the degree of adversarial noise increases (from 0 to 6 with step size 2), the clusters of different classes are less discriminative. In contrast to VAT, our proposed MAT obtains much more discriminative features on different levels of adversarial perturbation.
2) Visualization of Adversarial Examples: In this subsection, we visualize some adversarial examples generated by both our proposed method and the other comparison adversar-ial methods. To visualize the adversarial examples, we directly train the deep neural network with original images and then calculate the worst perturbation through back propagation. For better visualization, the image up-sampling was applied.
Specifically, we compare the adversarial examples generated by our method (MAT), VAT, and traditional adversarial learning on MNIST, SVHN, and CIFAR10. Figure 6-8 show several adversarial examples (generated by different methods) from the three datasets. It is interesting to note that after the adversarial training given by MAT, the original images tend to be morphed into some other categories. Particularly, digit "3" was changed into "8", "7" was changed to "9", and "9" was changed to "0" on MNIST (see Figure 6). Similar cases can also be observed in Figure 7 where the worst perturbation generated by MAT tends to change the number "6" to number "8" (the 1st image in the 2nd row of Figure ??), the number "3" to "5" (the 4th image in the 2nd row of Figure 7), and numbers "66" to "00" (the 8th image in the 2nd row of Figure 7). In comparison, the other adversarial methods also appear to generate adversarial examples in a same sense but not as obvious as MAT. This may partly explain why the adversarial examples would be difficult to be recognized by many traditional deep neural networks. Furthermore, generating the most challenging examples (being similar to or even changed to other categories), MAT indeed leads to the most serious adversarial attacks that deep neural networks hardly recognize. Finally, it is observed that the major changed areas occur on the number strokes, which can be considered as the "manifold" of these digit images. This implies that our MAT training method could truly generate adversarial attacks in the manifold.
Last, we also show in Figure 8 adversarial examples on CIFAR-10 generated from various methods. Unfortunately, very different from MNIST and SVHN, it appears that the resulting examples give no obvious clues on how the images would be changed. This might be explained by the fact that general objects are sufficiently more complicated than digits. We would leave the topic of generating explainable adversarial examples in the future.
V. CONCLUSION We present the Manifold Adversarial Training (MAT), a novel method to smooth the distributional manifold in the latent space. Compared with other adversarial training methods, our proposed MAT learns adversarially a robust feature representation in the latent space, making the latent space both informative and discriminative. Specifically, we first represent the latent features with Gaussian mixtures. We then define the smoothness of distributional manifold based on the KL-divergence between Gaussian mixtures for the original and adversarial examples. We implemented MAT on MNIST, CIFAR-10 and SVHN in both supervised and unsupervised tasks. The results showed that our proposed model is much better than those state-of-the-art methods.