We propose a new representation, one-pixel signature, that can be used to reveal the characteristics of the convolution neural networks (CNNs). Here, each CNN classifier is associated with a signature that is created by generating, pixel-by-pixel, an adversarial value that is the result of the largest change to the class prediction. The one-pixel signature is agnostic to the design choices of CNN architectures such as type, depth, activation function, and how they were trained. It can be computed efficiently for a black-box classifier without accessing the network parameters. Classic networks such as LetNet, VGG, AlexNet, and ResNet demonstrate different characteristics in their signature images. For application, we focus on the classifier backdoor detection problem where a CNN classifier has been maliciously inserted with an unknown Trojan. We show the effectiveness of the one-pixel signature in detecting backdoored CNN. Our proposed one-pixel signature representation is general and it can be applied in problems where discriminative classifiers, particularly neural network based, are to be characterized. Recent progress in designing convolutional neural network architectures (LeCun et al., 1989; Krizhevsky et al., 2012; Szegedy et al., 2015; He et al., 2016; Xie et al., 2017) has contributed, in part, to the explosive development in deep learning (LeCun et al., 2015; Goodfellow et al., 2016) . Convolutional neural networks (CNN) have been adopted in a wide range of applications including image labeling (Long et al., 2015; Ronneberger et al., 2015; Xie & Tu, 2015) , object detection (Girshick et al., 2014; Ren et al., 2015) , low-level image processing (Xie et al., 2012; Dosovitskiy et al., 2015a; Kim et al., 2016) , artistic transfer (Gatys et al., 2016) , generative models (Goodfellow et al., 2014a; Dosovitskiy et al., 2015b; Lee et al., 2018) , image captioning (Xu et al., 2015) , and 2D to 3D estimation/reconstruction Wu et al., 2017) . Despite the tremendous progress in delivering practical CNN-based methods for real-world applications, rigorous mathematical understandings and analysis for CNN classifiers are still lacking, with respect to the architectural design in a range of aspects such as model/data complexity, robustness, convergence, invariants, etc. (Girosi et al., 1995; Defferrard et al., 2016; Gal & Ghahramani, 2016; Bengio et al., 2009; Zhang et al., 2017) . Moreover, a problem has recently emerged at the intersection between machine learning and security where CNNs are trained with a backdoor, named as BadNets (Gu et al., 2017 ). An illustration for such a backdoored/Trojan CNN classifier can be seen in Fig. 1 .b . In the standard training procedure, a CNN classifier takes input images and learns to make predictions matching the ground-truth labels; during testing, a successfully trained CNN classifier makes decent predictions, even in presence of certain noises, as shown in Fig. 1 .a . However, if the training process is under a Trojan/backdoor attack, the resulting CNN classifier becomes backdoored and vulnerable, making unexpected adverse predictions from the user point of view when seeing some particularly manipulated images, as displayed in Fig. 1 .b . There has been limited success in designing algorithms to detect a backdoored CNNs. We develop one-pixel signature and make the following contributions. • To unfold CNN classifiers to perform e.g. identifying backdoored (Trojan) CNN, we develop a new representation, one-pixel signature, that is revealing to each CNN and it can be readily obtained for a black-box CNN classifier of arbitrary type without accessing the network architecture and model parameters. • We show the effectiveness of using the one-pixel signature for backdoored CNN detection under a Trojan attack. Various network architectures including LeNet (LeCun et al., 1989) , AlexNet (Krizhevsky et al., 2012) , ResNet (He et al., 2016) , DenseNet (Huang et al., 2017) , and ResNeXt (Xie et al., 2017) are studied. • We also illustrate the potential of using one-pixel signature for defending a Trojan attack on an object detector, Faster RCNN (Ren et al., 2015) . • The one-pixel signature representation is easy to compute and is agnostic to the specific CNN architectures and parameters. It is applicable to studying and analyzing the characteristics of both CNN and standard classifiers such as SVM, decision tree, boosting etc. (a) CNN trained regularly (b) CNN trained with a backdoor displays a backdoored CNN, denoted as CNNT rojan, which is trained maliciously by inserting a "virus" pattern (a star) to a training sample and forcing the classification to a wrong label. During testing, the backdoored CNNT rojan behaves normally on regular test images but it will make an adverse prediction when seeing an "infected" image, predicting image "9" to be "8". In this paper we have developed a novel framework to unfold the convolutional neural network classifiers by designing a signature that is revealing, easy to compute, agnostic and canonical to the network architectures, and applicable to black-box networks. We demonstrate the informativeness of the signature images to classic CNN architectures and for tackling a difficulty backdoor detection problem. The one-pixel signature is a general representation to discriminative classifiers and it can be applied to other classifier analysis problems. for j from 0 to n − 1 do 7: for v from 0 to 1; step=1/V do 10: for k from 0 to K-1 do 12:
Cnvolutional neural networks characterization for backdoored classifier detection and understanding.
{"src": "scitldr-AIC_train", "paper_id": "BkxDKREtPS"}