---
abstract: 'Nowadays, deep learning technology is growing faster and shows dramatic performance in computer vision. However, it turns out that the deep learning model is highly vulnerable to small perturbation called an *adversarial attack*. So far, although many of the *defense mechanism* has been proposed to mitigate the effect of the adversarial attack, all of them are under rigorous assumptions. However, our approach is not tied up any assumptions since our insight stems from the *Tensor Decomposition*. In this paper, we experimentally demonstrated that decomposing the tensor would be an effective countermeasure against several adversarial attacks. We conducted experiments with well-known benchmarks such as MNIST, CIFAR-10, and ImageNet dataset. Our experimental results show that this simple method has capable of having attack resilience and robustness against adversarial attacks. To the best of our knowledge, this is the first approach to leverage the tensor decomposition as a defense mechanism. We hope that leveraging the tensor decomposition becomes a universal approach to solve inherent corner cases of deep learning models.'
author:
- 'Seungju Cho, Tae Joon Jun, Mingu Kang, Daeyoung Kim'
bibliography:
- 'egbib.bib'
title: Applying Tensor Decomposition for the Robustness against Adversarial Attack
---

Introduction
============

Over the past several years, advances in deep neural networks (DNNs) have widely expanded the ability of what the machine can deal with. Especially, DNNs have achieved remarkable successes for image classification [@krizhevsky2012imagenet; @sermanet2013overfeat] and it even goes beyond human capability [@he2016deep]. With this performance, deep learning technology has started to be applied to various areas. However, some papers [@szegedy2013intriguing; @goodfellow2014explaining; @carlini2017towards; @kurakin2016adversarial; @kurakin2016adversarial_scale; @moosavi2016deepfool; @Eykholt_2018_CVPR; @Chen2017EADEA] proved that even DNNs can be easily fooled by small changes to input that is imperceptible to a human eye. According to these studies, carefully crafted perturbations to the vision-based applications can induce systems to behave in unexpected ways. Indeed, this is small enough to be inconspicuous, but some researches show that its influence might be more than expected since even state-of-the-art models get an almost zero-classification accuracy under [@carlini2017towards].

Considering the deep learning models do not hesitate whenever judge the output, it might cause crucial accidents. For instance, Fig. \[fig:AdvExample\] represents the adversarial examples in the image classification task. Although all of the images can be seen as an ostrich by human visible intuition, deep learning model outputs clearly different labels due to lack of such intuition. From a more theoretical perspective, misclassification occurs when the adversarial perturbations cross the decision boundary, but the existing classifier has no such intuition that can ward it off. Now, the corner case of DNNs which have been alluded to adversarial attacks is getting pervasive and being more sophisticated. As a result, the vulnerability of adversarial attacks hinders its adoption for some safety-critical system and also security-sensitive application, including an autonomous-driving car [@Eykholt_2018_CVPR; @sitawarin2018darts].

Since the advent of such adversarial attacks, many researchers or vendors have paid significant attention to adversarial examples. This is because they might not want to go through all the risks of their applications or models. They might as well choose to verify the robustness rather than take risks. However, the resistance against adversarial examples renders another challenge, for no method can be a cure-all against adversarial attacks. To make up for corner cases of DNNs, several studies [@goodfellow2014explaining; @szegedy2013intriguing; @meng2017magnet; @papernot2016distillation; @song2017pixeldefend; @jia2019comdefend; @Liao_2018_CVPR; @xie2019feature] have proposed the defense mechanism against adversarial attacks to mitigate the potential of the risk by adversary. These defense mechanisms can be viewed as two main approaches: (1) changing the model itself, which can improve the robustness by training with adversarial examples, e.g., *adversarial training* [@goodfellow2014explaining; @szegedy2013intriguing; @tramer2017ensemble], (2) preprocessing the inputs to diminish the effect of adversarial noise, e.g., *Magnet, Comdefend, PixelDefend, Defense-GAN, HGD, etc.* [@meng2017magnet; @jia2019comdefend; @song2017pixeldefend; @samangouei2018defense; @Liao_2018_CVPR; @xie2019feature]. However, (1) are designed to deal with specific adversarial attack strategies in mind, so generalization is likely to be restricted. It implies that the models using this method may be vulnerable to another attack optimized with such attack strategies. On the other hand, (2) utilize models with a vast amount of legitimate data to purify the inputs itself, instead of assuming some attack strategies. The main point of (2) is to measure the distance between the inputs and manifold of the legitimate images, and then approximate or guide the adversarial images closer to the manifold of the legitimate images. To purify the inputs, therefore, a well-generalized model should be required to assure the performance. Given that the adversarial images occupy the low probability region trained with legitimate samples [@song2017pixeldefend], poorly generalized models cannot ensure that the aforementioned approaches get a good result. Also, another attack technique might be created by an adversary who knows the model’s structure. In a nutshell, as these approaches are likely to be a temporary expedient, the universal defense approaches which can cover a myriad of risk should be explored. Here, we propose a novel intuition for deep learning models, which can make the model universally robust. To the best of our knowledge, this is the first approach to explore the defense in terms of the universal point of view. Our approach leverages the potential power of the tensor decomposition to diminish the effect of adversarial noise by using the reconstructed images as an input of a deep learning model. The reconstructed inputs are fed into the classifier, and we experimentally demonstrate that such simple preprocessing could be an effective countermeasure against the adversarial attack. On MNIST, a degradation of top-1 accuracy on adversarial example is less than 1% against four adversarial attacks and less than 10 % on CIFAR-10 and ImageNet. This result outperforms recent defense mechanisms [@Liao_2018_CVPR; @jia2019comdefend]. To ensure that deep learning applications extend their potential of utilization toward other domains, it would be better to take into account the robustness of those applications. If you want to avoid cherry-picking doubt and make the model more general across a variety of risks including adversarial attacks, our insight would be an interesting candidate. Our contribution is as follows:

1.  **High Compatibility.** Our approach leverages the tensor decomposition for preprocessing the inputs which might have been affected by the adversary. We do not assume anything such as attack strategies or classifiers, just use an input as a reconstructed input by using the tensor decomposition method, which indicates that our proposed method can be relatively easy to utilize and be applied to whatever the classifier is.

2.  **Efficient Engineering Complexity.** As we mentioned above, tensor decomposition just depends on what the input it is, so it is not tied up with the attack strategies or classifiers. Therefore, we do not need to focus on how the model classifies the input since tensor decomposition is free from the model dependency. It implies that retraining the model or augmenting the training data could no longer be required. It requires only processing time to reconstruct inputs. Even more, the processing time is negligible.

3.  **Integrity of the inputs.** When it comes to the reconstruction process, some information that in charge of the important role might be lost. Although state-of-the-art approaches [@Liao_2018_CVPR; @jia2019comdefend] have gotten remarkable performance, their proposed model degrades the performance with even the clean images. This is some kind of a trade-off. It thus makes it difficult to apply defense mechanisms. However, tensor decomposition could incur less adverse effects on the clean images, and ensure its performance even at the high-dimension dataset such as ImageNet.

Related work
============

#### **Adversarial Attacks.** 

Szegedy *et al.* found the existence of adversarial perturbation that breaks the image classifier thorough solving adversarial optimization problem [@szegedy2013intriguing; @goodfellow2014explaining]. They show the model accuracy is dropped even though the perturbed image looks similar to human eyes. Goodfellow *et al.* [@goodfellow2014explaining] uses the sign of the gradient of input with respect to the loss function of the target model. This method is called Fast Gradient Sign Method (FGSM) since it updates input once. With a similar idea, [@kurakin2016adversarial] uses FGSM in an iterative way. Chen *et al.* [@Chen2017EADEA] leverages $L_{1}$ distortion to generate effective adversarial examples and improve the attack transferability which refers to the attack success rate using the adversarial examples which come from the substitute models. In other words, high transferability implies that the performance of the target model might depreciate even without the knowledge about the model, i.e., *black box attack*. Carlini and Wagner [@carlini2017towards] changes the optimization problem defined in [@szegedy2013intriguing] for achieving more powerful attack. [@moosavi2016deepfool] measures the minimum size required for the attack. They approximate the decision boundary of the model and update input repeatably until the model misclassifies it. Besides the image classification task, [@Eykholt_2018_CVPR; @sitawarin2018darts] demonstrated that adversarial attacks can be applied beyond the digital space, so security concerns could arise in even physical space such as the autonomous-driving car.

#### **Adversarial Defense.**

To counter adversarial attacks, some works trained the model with adversarial examples to ensure that the model has a resilience against those adversarial examples, which have been called *adversarial training*. During the process of training, they generate adversarial images for improving the performance. Although it works, it depends on the particular adversarial data used in the training process. For instance, [@kurakin2016adversarial_scale] shows their approach is robust in the simple attack, but not in a more sophisticated attack. In addition, it has an engineering penalty since it requires retraining the model. If it takes longer to create an adversarial example, it will take more time to retrain the model. Instead of using the data augmentation, methods to change the model itself were also proposed [@papernot2016distillation]. They change the objective function of the problem for obtaining the robustness. However, this approach also has to retrain the model, so it also boils down to increasing the engineering complexity.

In recent years, several papers [@meng2017magnet; @jia2019comdefend; @song2017pixeldefend; @Liao_2018_CVPR; @xie2019feature; @samangouei2018defense] preprocess the inputs before putting into the classifier. They propose the model which serves the direction to approximate the distribution of the adversarial images as close as possible to the decision boundary. All of the methods require a well-generalized defense model, so the even clean images could be affected when the model is poorly generalized. It results in damage to the integrity of the inputs. To guarantee the integrity of the model, all of them require well-generalized classifiers to detect if the input is adversarial or approximate the manifold of legitimate samples. Our approach is similar to those approaches in terms of the preprocessing, yet differentiation is our method does not need any premises, including the detector or well-generalized models. Consequently, our method does not hurt the performance in terms of integrity.

Background
==========

Adversarial Attack
------------------

Basically, all of the attacks use the gradient of data with respect to the loss function of the target model. In this section, we briefly review the basic method of adversarial attack.

**Fast Gradient Sign Method (FGSM):** The *FGSM* is proposed by [@goodfellow2014explaining]. It is a simple and effective attack method. The image $X$ is perturbed as follows. $$X = X + \epsilon \cdot  | sign(\triangledown_{X} l(X,y_{true})) |$$ Where $\epsilon$ is a magnitude of noise and $l(X,y_{true})$ is a loss with respect to the true label of the image. It adjusts input X by adding a sign of the gradient of X. It increases the loss function of the target model so that the model misjudges the adjusted input. Since it updates input X once, it is also called *single-step* method.

**Basic Iterative Method (BIM):** The *BIM* is a repetitive version of FGSM [@kurakin2016adversarial]. it is a more powerful attack method compared to the FGSM. And it is also called *Iterative FGSM*. It uses the following equations: $$X_{t+1} = \text{clip}_{X,\epsilon}(X_{t} + \alpha  \cdot   | sign(\triangledown_{X} l(X_{t},y_{true})) |)$$ Where $X_{0} = X$, $\alpha$ is a step size for adjusting $X_t$ , and clip function ensures that $X_t \in (X-\epsilon,X+\epsilon )$ for all $t$. It is also called *multi-step* method. Here $\alpha$ is the $1$ in the scale of 0 to 255 in the original paper.

**DeepFool:** DeepFool attack approximates the decision boundary of a classifier, and measure the minimal perturbations that are sufficient to fool the classifier [@moosavi2016deepfool]. For the affine multiclass classifier, they calculate the distance $d$ as follows. $$d = \frac{|f_k(x) - f_l(x)|}{w'} ~ \text{where}~  w' = \|\nabla f_k(x) - \nabla f_l(x) \|_2$$ Here $f_k$ and $f_l$ are classifier for $k$ and $l$-th class. Similar to BIM, they update $x$ in an iterative way. For nonlinear classifiers, they approximate the linear boundary and find the distance to fool the nonlinear classifier.

**Carlini & Wagner (C&W):** Carlini and Wagner [@carlini2017towards] define an optimizaion problem to find an adversarial example. They define following optimizaion formulation. $$\text{min}_{\delta} ~ D(x,x+\delta) + c \cdot f(x+\delta)$$

Here $D$ is a distance metric to measure the distance between the clean image and adversarial image. $f(\cdot)$ is an objective function to control the result of original classifier $C$. C&W attack is one of the most powerful attacks in this literature. We visualize the adversarial image generated by each method in Fig. \[fig:advs\]

[0.19]{} ![Visualization of adversarial examples on each method. The pre-trained Resnet101 [@he2016deep] model classifies clean image as a *monarch*, image with FGSM as a *sea slug*, image with BIM as a *Doberman*, image with DeepFool as a *hornbill* and image with C&W as a *longicorn*. However, we can check that theses images seem to the same in the human eye[]{data-label="fig:advs"}](Images/clean.png "fig:"){width="\textwidth"}

[0.19]{} ![Visualization of adversarial examples on each method. The pre-trained Resnet101 [@he2016deep] model classifies clean image as a *monarch*, image with FGSM as a *sea slug*, image with BIM as a *Doberman*, image with DeepFool as a *hornbill* and image with C&W as a *longicorn*. However, we can check that theses images seem to the same in the human eye[]{data-label="fig:advs"}](Images/FGSM.png "fig:"){width="\textwidth"}

[0.19]{} ![Visualization of adversarial examples on each method. The pre-trained Resnet101 [@he2016deep] model classifies clean image as a *monarch*, image with FGSM as a *sea slug*, image with BIM as a *Doberman*, image with DeepFool as a *hornbill* and image with C&W as a *longicorn*. However, we can check that theses images seem to the same in the human eye[]{data-label="fig:advs"}](Images/bim.png "fig:"){width="\textwidth"}

[0.19]{} ![Visualization of adversarial examples on each method. The pre-trained Resnet101 [@he2016deep] model classifies clean image as a *monarch*, image with FGSM as a *sea slug*, image with BIM as a *Doberman*, image with DeepFool as a *hornbill* and image with C&W as a *longicorn*. However, we can check that theses images seem to the same in the human eye[]{data-label="fig:advs"}](Images/deepfool.png "fig:"){width="\textwidth"}

[0.19]{} ![Visualization of adversarial examples on each method. The pre-trained Resnet101 [@he2016deep] model classifies clean image as a *monarch*, image with FGSM as a *sea slug*, image with BIM as a *Doberman*, image with DeepFool as a *hornbill* and image with C&W as a *longicorn*. However, we can check that theses images seem to the same in the human eye[]{data-label="fig:advs"}](Images/carlini.png "fig:"){width="\textwidth"}

Tensor Decomposition
--------------------

A *tensor* is a multi-dimensional array. For instance, the color image is a tensor consists of height, width, and the color channel. A tensor decomposition method decomposes a tensor into low dimension tensors. The *CANDECOMP/PARAFC* [@carroll1970analysis; @harshman1970foundations] decomposition approximates a tensor $X$ as a sum of the outer product of the tensor belonging to each dimension as Fig. \[fig:cp\]. We refer to this as a *CP* decomposition.

Let $X \in \mathbb{R}^{I \times J \times K}$, $a_i \in \mathbb{R}^{I}, b_i \in \mathbb{R}^{J} \ \text{and}\ c_i \in \mathbb{R}^{K}$ for $i = 1,\dots,r$. Here $r$ is the number of components and it is a hyperparameter. If $r$ is small, tensor$X$ is approximated into low dimension tensor. So we call deciding $r$ as choosing the dimension for convenience. Then $x_{ijk}$ is approximated as follows. $$x_{ijk} \approx \sum_{l = 1}^{r} a_{il}b_{jl}c_{kl}$$

The *Tucker* decomposition [@tucker1963implications; @tucker1966some] is another way to decompose a tensor. It is a kind of higher order principal component analysis [@kolda2009tensor]. It decomposes tensor as a core tensor and factor tensors as Fig. \[fig:tucker\].

Let $X \in \mathbb{R}^{I \times J \times K}$, $A \in \mathbb{R}^{I \times P}, B \in \mathbb{R}^{J \times Q} , C \in \mathbb{R}^{I \times R}  \text{and} \ G \in \mathbb{R}^{P \times Q \times R} $ for $i = 1,\dots,I, j = 1,\dots,J,k = 1,\dots,K$. Here the size of the core tensor $P,Q$ and $R$ are the number of the components and it is hyperparameter. When the size of the core tensor is fixed, the size of the factor tensor is decided according to the size of the core tensor. Similar to the number of the components of *CP* decomposition, we call deciding the size of the core tensor as a choosing the dimension. Then $x_{ijk}$ is approximated as follows.

$$x_{ijk} \approx \sum_{p = 1}^{P}\sum_{q = 1}^{Q}\sum_{r = 1}^{R} g_{pqr}a_{ip}b_{jq}c_{kr}$$

Method
======

To mitigate the effect of the adversarial attacks, our insight stems from the tensor decomposition. In this section, all the paragraphs that describe revolve around how we can apply this magic, i.e., tensor decomposition, as a defense mechanism against the adversarial attacks.

Tensor decomposition as a preprocessing
---------------------------------------

As you can see in Fig. \[fig:advs\], adversarial examples are too sophisticated to be recognized by human senses, including well-generalized deep learning models. To prevent the potential threat, our model simply uses the reconstructed image from the tensor decomposition as an input. We conjecture that the effect of the adversarial perturbations could be reduced by approximating the tensors toward the low dimension. To cast light on our hypothesis, we conduct brief experiments based on the visual sense. Fig. \[fig:noise\] represents the examples of the noise. Intuitively, adversarial noise crafted by the FGSM seems to distinguishable from others, while the rest of them are relatively similar to each other. In other words, the tensor decomposition can transform adversarial noise into random noise, e.g., gaussian noise. We can say that the tensor decomposition has an ability to purify the adversarial noise in this regard. Even though such random noise might also degrade the performance, it would be no worse than original adversarial noise considering they are crafted by adversarial intend.

[0.24]{} ![Visualization of example noises. (a) is the adversarial noise crafted by FGSM. (b) is the gaussian noise, (c),(d) are reconstructed images by using *Tucker*, *CP* decomposition respectively[]{data-label="fig:noise"}](Images/adv_noise.png "fig:"){width="\textwidth"}

[0.24]{} ![Visualization of example noises. (a) is the adversarial noise crafted by FGSM. (b) is the gaussian noise, (c),(d) are reconstructed images by using *Tucker*, *CP* decomposition respectively[]{data-label="fig:noise"}](Images/gaussian.png "fig:"){width="\textwidth"}

[0.24]{} ![Visualization of example noises. (a) is the adversarial noise crafted by FGSM. (b) is the gaussian noise, (c),(d) are reconstructed images by using *Tucker*, *CP* decomposition respectively[]{data-label="fig:noise"}](Images/cp_noise.png "fig:"){width="\textwidth"}

[0.24]{} ![Visualization of example noises. (a) is the adversarial noise crafted by FGSM. (b) is the gaussian noise, (c),(d) are reconstructed images by using *Tucker*, *CP* decomposition respectively[]{data-label="fig:noise"}](Images/tucker.png "fig:"){width="\textwidth"}

![We show a change of reconstructed images using *CP* and *Tucker* decomposition by varying the value of the dimension (The value is getting higher order by left $\rightarrow$ right)[]{data-label="fig:gradation"}](Images/cp_gradation.png){width="\textwidth"}

![We show a change of reconstructed images using *CP* and *Tucker* decomposition by varying the value of the dimension (The value is getting higher order by left $\rightarrow$ right)[]{data-label="fig:gradation"}](Images/tucker_gradation.png){width="\textwidth"}

Based on this light, we utilize *CP* and *Tucker* decomposition methods to verify how effective these methods actually are under various attack strategies. Before the main experiments, both methods require to set the dimension, e.g., $r$ for the *CP* and $P,Q$, $R$ for the *Tucker*. As the dimension of the tensor increases, the quality of the reconstructed image gets better as shown in Fig. \[fig:gradation\]. The high quality of the reconstructed image is not always better, however, so the dimension needs to be decided in a heuristic manner. We thus studied the ablation study to find out the effective hyperparameters under *CP* and *Tucker* and consider two kinds of factors to decide the hyperparameters, accuracy and time complexity. We randomly sampled 1,000 images from CIFAR-10, and then generate the adversarial images by applying FGSM, BIM, DeepFool, and C&W, respectively. As follow, those images are reconstructed by *CP* and *Tucker* decomposition. Finally, we measured the accuracy and time complexity using the reconstructed images.

[0.48]{} ![ (a) Accuracy with respect to dimension (b) Mean time for processing single image with respect to dimension[]{data-label="fig:graph"}](Images/accuracy.png "fig:"){width="\textwidth"}

[0.48]{} ![ (a) Accuracy with respect to dimension (b) Mean time for processing single image with respect to dimension[]{data-label="fig:graph"}](Images/time.png "fig:"){width="\textwidth"}

The result summarized in Fig. \[fig:graph\]. While accuracy increased as dimensions increased by the middle, accuracy tends to decrease gradually. And the processing time increases as dimension increases. So we decide to use 40 % of the original dimension. For instance, the size of the image is 32 by 32 in the CIFAR-10 dataset. In the case of *CP* decomposition, the size of the three tensors in Fig. \[fig:cp\] will be 32,32 and 3. So we choose rank $r = 8$. We use a similar argument when choosing the size of the core tensor of *Tucker* decomposition. In particular, we don’t compress the channel dimension $3$ in *Tucker* decomposition since we don’t want to lose color information. Thus, the size of the core tensor of the *Tucker* decomposition is in the form of height, width and $3$. We use an open-source library [@kossaifi2019tensorly] for each decomposition method.

Denoise Autoencoder as a supplement
-----------------------------------

We should consider one more thing before putting the input value into the classifier. When it comes to reconstructing the images, we should consider the loss of information as it might affect the classification result. If the input is clean images, it would work even worse. To diminish the adverse effect from that point, we add denoise autoencoder into the procedure. Our method is based on a coarse-to-fine approach. Through the reconstructed inputs from decomposed tensors, we remove the coarse-grained adversarial features. We expect that some fine-grained features that might be lost by the coarse-grained approach—which is more likely to occur in high-dimension, could be compensated pass through the denoise autoencoder. Equipped with this approach, we set up the denoise autoencoder architecture as follows. The numerical value in Table \[table:autoencoder\] stands for input channel and output channel respectively. And the filter size is $3 \times 3$.

  ---------- ---- ---- ---------- ---- ---- -- -- -- -- -- -- --
  MNIST                  CIFAR                                
  Conv,ELU    1   2     Conv,ELU  3     6                     
  Conv,ELU    2   4     Conv,ELU  6     12                    
  Conv,ELU    4   8     Conv,ELU  12    24                    
  Conv,ELU    8   16    Conv,ELU  24    48                    
  Conv,ELU    16  32    Conv,ELU  48    96                    
  Conv,ELU    32  16    Conv,ELU  96    48                    
  Conv,ELU    16  8     Conv,ELU  48    24                    
  Conv,ELU    8   4     Conv,ELU  24    12                    
  Conv,ELU    4   2     Conv,ELU  12    6                     
  Conv,ELU    2   1     Conv,ELU  6     3                     
  ---------- ---- ---- ---------- ---- ---- -- -- -- -- -- -- --

  : Architecture of autoencoder trained with each dataset[]{data-label="table:autoencoder"}

We utilize the CIFAR-10 images at the RGB scale and MNIST image at the grayscale. We set the learning rate to $1e-4$ and used Adam [@kingma2014adam] as an optimizer. And we use mean square error (MSE) for loss function. For both models, we train autoencoder for 10 epochs. We henceforth denote autoencoder as *AE*.

Overall architecture
--------------------

We describe the overall architecture in detail. Fig. \[fig:architecture\] represents the overall flow of our proposed method. First, we approximate the input image via the tensor decomposition method. The inputs could be adversarial images or clean images. Our method does not spend time deciding whether the input is adversarial or not. That’s the reason why our model does not require a well-generalized model. In other words, whatever the input is, our model splits the input into several tensors based on *CP* or *Tucker* decomposition, and then reconstruct them. As follows, the reconstructed images are passed through the denoise autoencoder, which might compensate for losing the information that may in charge of an important role in that image. Note that our method does not have a model dependency, so it can be applied in conjunction with every classifier.

Experiment
==========

Dataset
-------

For the MNIST, CIFAR-10 data, we test on the full test data, which are composed of 10,000 images on MNIST and 50,000 images on CIFAR-10. For ImageNet, we select randomly 1,000 images as similar setting [@jia2019comdefend; @kurakin2018adversarial]. Since our proposed method decomposes the input whichever clean image or adversarial image, we tested on both clean images and adversarial images.

Evaluation
----------

We measure the top-1 accuracy on clean images and adversarial images on each dataset. For MNIST, we use a simple model consists of two convolutional layers. For CIFAR-10 and ImageNet, we basically use pre-trained Resnet101 [@he2016deep]. In particular, for CIFAR-10, we finetune pre-trained Resnet101 for 10 classes. We use FGSM, BIM, DeepFool and C&W attack methods. For the distance metric, a related research area mainly uses $L_{\infty}$ and $L_2$ norm [@carrara2018adversarial; @jia2019comdefend]. In detail, we use $L_{\infty}$ for FGSM, BIM, and DeepFool attack. And for the C&W attack, we use a $L_2$ metric. We try to find small perturbation when applying the adversarial attack since the noise is visible when the perturbation is not small enough. We generate adversarial images by using open source library Foolbox [@rauber2017foolbox]. In detail, we try 100 epsilons from 0 to 1 for FGSM. For BIM, we set 5 as a number of iteration. And for DeepFool, we set 50 as a maximum number of steps and set 50 as a maximum iteration number of C&W. And we measure the pre-processing time for calculating the additional time consuming for the proposed method. Also, we compare our results to other state-of-the-art defense models. For a fair comparison, we compare the ratio between the accuracy of the clean image and the adversarial image since the accuracy of a clean image is a little bit different depending on the setting.

Results
-------

We achieve remarkably high accuracy against adversarial attacks. In most cases, the *CP* is better than *Tucker* decomposition method. In some case of ImageNet dataset, *Tucker* decomposition method is better than *CP*. For instance, when attack with FGSM and C&W method, the result was the best by using *Tucker* decomposition. And in the case of clean images, the accuracy reduction was about 1% on all datasets. It means that we do not harm the original model in a normal case which is the input image is clean. The autoencoder is highly effective on MNIST dataset. Although the autoencoder does not have much effect on clean images, it improves the performance of various adversarial attacks on MNIST dataset. In addition to MNIST dataset, there have been small performance improvements for other datasets by using the denoise autoencoder. The numerical results are summarized in Table \[table:MNIST\], \[table:CIFAR\] and \[table:Imagenet\].

  ------- ----------- ------- ------- ------- ---------- -------
  Model   Method      Clean   FGSM    BIM     DeepFool   C&W
          Original    99.06   0.00    0.21    0.00       0.00
          CP          99.01   71.62   95.42   81.68      88.29
          CP+AE       98.64   96.15   98.88   98.83      98.6
          Tucker      99.06   71.95   91.81   79.32      84.52
          Tucker+AE   98.06   95.26   98.65   98.54      98.05
  ------- ----------- ------- ------- ------- ---------- -------

  : Top-1 accuracy of each method on MNIST dataset[]{data-label="table:MNIST"}

  ------- ----------- ------- ------- ------- ---------- -------
  Model   Method      Clean   FGSM    BIM     DeepFool   C&W
          Original    98.87   0.0     0.0     0.0        0.0
          CP          97.96   92.52   92.49   94.67      93.63
          CP+AE       98.11   92.88   93.21   94.93      94
          Tucker      95.99   87.74   87.68   90.73      90.57
          Tucker+AE   96.00   88.44   88.56   91.28      91.2
  ------- ----------- ------- ------- ------- ---------- -------

  : Top-1 accuracy of each method on CIFAR-10 dataset[]{data-label="table:CIFAR"}

  ------- ----------- ------- ------- ------- ---------- ------
  Model   Method      Clean   FGSM    BIM     DeepFool   C&W
          Original    77.6    0.0     0.0     0.0        0.0
          CP          76.3    61.2    75.4    75.7       61.2
          CP+AE       74.8    62.9    75.00   75.5       62.1
          Tucker      76.4    65.00   73.4    73.8       65.3
          Tucker+AE   74.7    65.6    73.7    74.7       66
  ------- ----------- ------- ------- ------- ---------- ------

  : Top-1 accuracy of each method on ImageNet dataset[]{data-label="table:Imagenet"}

Even the DeepFool and C&W attacks are more accurate and powerful attack compared to the FGSM and BIM attacks, the accuracy after decomposition is higher than the case of FGSM and BIM attacks.

Comparison with other defense methods
-------------------------------------

We measure the ratio of accuracy on clean images and adversarial images generated by FGSM, BIM, DeepFool and C&W attack for a fair comparison. Here the $L_{\infty}$ is restricted to $8$ in $255$ scale. The defense ratio is defined as follows. $$\textit{Defense ratio} = \frac{\textit{Top-1 accuracy on adversarial images}}{\textit{Top-1 accuracy on clean images}}$$ We compare the performance of recent defense methods, HGD [@Liao_2018_CVPR] and Comdefend [@jia2019comdefend]. Fig. \[fig:bar\] shows the results. We select Resnet101 [@he2016deep] and Inception V3 (IncV3) [@szegedy2016rethinking] as base model. And we tested on 1,000 images from the ImageNet data. Our methods outperform in all cases compared to two recent defense methods. This result verifies that our method is effective. Moreover, our method does not depend on attack methods and the target model classifier, thus it can be easily combined with every model.

[0.48]{} ![ (a) Defense ratio of each attack method on Resnet101 (b) Defense ratio of each attack method on IncV3. Note that our method outperforms other defense methods[]{data-label="fig:bar"}](Images/bar_1.png "fig:"){width="\textwidth"}

[0.48]{} ![ (a) Defense ratio of each attack method on Resnet101 (b) Defense ratio of each attack method on IncV3. Note that our method outperforms other defense methods[]{data-label="fig:bar"}](Images/bar_2.png "fig:"){width="\textwidth"}

Time analysis
-------------

We measure the preprocessing time of each method. We pick randomly 1,000 images in MNIST,CIFAR-10, and ImageNet. And we calculate the average processing time per image. In most case, the *CP* decomposition requires more time compared to *Tucker* decomposition. In the case of MNIST and CIFAR-10, the time required to reconstruction is similar in both cases. However, In the case of ImageNet, the *CP* decomposition takes about 10 times more than the *Tucker* method. Table \[table:time\] summarizes preprocessing time of each dataset on each method.

[\*5c]{} dataset&CP&CP+AE&Tucker&Tucker + AE\
MNIST & 0.005 & 0.006 & 0.003 & 0.004\
CIFAR-10 & 0.1052 & 0.1161 & 0.01268 & 0.015\
ImageNet & 1.07 & 1.18 & 0.1566 & 0.17\

White box scenario
------------------

In the white box scenario, we should assume the adversary knows full defense mechanism according to [@carlini2017adversarial]. In our method, note that the input image is always decomposed and reconstructed, and the decomposed components are always started from the random tensor. In detail, the component tensors of each decomposition method initialized to random tensor and then trained to approximate the original tensor. Thus, the input is always random tensor and the original image is a label itself like unsupervised learning. Therefore, there are no fixed weights, so the adversary can not generate adversarial examples concerning the tensor decomposition method. Hence, our propose method is robust on the white box attack scenario.

Conclusions
===========

In this work, we verify the tensor decomposition is a simple and powerful method for purifying the adversarial perturbation. When we combine denoise autoencoder with the tensor decomposition method, the proposed method achieves higher accuracy against adversarial attacks. We experiment with our method against various adversarial attacks such as DeepFool and C&W attacks and discuss why this method is robust in the white box scenario.

Our intuition applying tensor decomposition into the adversarial attack is as follows. Since the adversarial perturbation is so small that it is hard to catch a difference, such a small perturbation would be removed by approximating the image tensor using low dimensional tensors. Since there is no straightforward algorithm to choose the dimension of the component tensors of the *CP* and *Tucker* decomposition, finding the best dimension remains for future work. Also, establishing a theoretical base why tensor decomposition is robust against adversarial attack is left to our future work.
