Adversarial examples remain an issue for contemporary neural networks. This paper draws on Background Check , a technique in model calibration, to assist two-class neural networks in detecting adversarial examples, using the one dimensional difference between logit values as the underlying measure. This method interestingly tends to achieve the highest average recall on image sets that are generated with large perturbation vectors, which is unlike the existing literature on adversarial attacks . The proposed method does not need knowledge of the attack parameters or methods at training time, unlike a great deal of the literature that uses deep learning based methods to detect adversarial examples, such as , imbuing the proposed method with additional flexibility. Adversarial examples are specially crafted input instances generated by adversarial attacks. The term was introduced by BID23 in the context of image classification. These attacks generate, or manipulate data, to achieve poor performance when classified by neural networks, which poses existential questions about their usage in high stakes security critical applications. Since they were introduced, there have been many papers that have introduced novel attack methods and other papers that attempt to combat those attacks. For instance, BID5 introduces the fast gradient sign method (FGSM), and BID20 proposes a method based on modifying the gradient of the softmax function as a defence. Adversarial attacks can be identified into various classes such as white box and black box, where in the former, the attack has full knowledge of all model parameters. Examples created by these attacks can be false positives or false negatives. In the case of images, they can be nonsensical data (e.g. noise classified as a road sign) or clear cut (e.g. a visually clear cat, classified as a road sign). These attacks can be non-targeted or targeted such that the classifier chooses a specific class for the adversarial example. Various adversarial defences exist, some based on deep learning techniques and others on purely distributional techniques. Similar work on adversarial defences has been done by BID6, in which the network is trained on specific attack types and parameters with an additional outlier class for adversarial examples. A multi-dimensional statistical test over the maximum mean discrepancy and the energy distance on input features is then used to classify instances as adversarial. Other work has been done by BID0, where Gaussian Processes are placed on top of conventional convolutional neural network architectures, with radial basis kernels, imbuing the neural network with a way of understanding its own perceptual limits. The authors find that the network becomes more resistant to adversarial attack. The work that follows continues in a similar vein to both of these methods. Some methods such as BID14 use sub-units of deep learning architectures to detect adversarial instances. Calibration is a technique of converting model scores, normally, through application of a post processing function, to probability estimates. Background Check is a method to yield probability estimates, via a set of explicit assumptions, in regions of space where no data has been observed. In this work, Background Check is useful in producing calibrated probabilities for adversarial data that often exists in regions where no training and test data has been seen. Reliable probability estimates can then be measured by calibration and refinement loss. Various calibrating procedures exist such as binning, logistic regression, isotonic regression and softmax. BID8 demonstrates the logistic function is optimal when the class-conditional densities are Gaussians with unit variance. Softmax extends this to multi-variate Gaussian densities with unit variance. Calibration of neural network models has been performed by BID7, using a method called Temperature Scaling, that modifies the gradient of the softmax function allowing softmax to calibrate densities with non-unit variance. The authors perform this calibration after noticing that calibration loss for neural networks has increased in recent years. When adversarial attacks against neural networks are brought into perspective, a problem arises for existing calibration techniques, which is the question of mapping adversarial logit scores to reliable probability estimates (which should be zero for a successful adversarial attack). In this work, a method is demonstrated that uses Background Check to identify adversarial attacks. A classifier is said to be well calibrated, if, as the number of predictions approaches infinity, the proportion of outcomes given probability p, occur p fraction of the time. Denoting x 1, x 2,..., x n as data-set instances and y 1, y 2,..., y n as their corresponding ground truth class labels, a scoring classifier s = f (x i) has a calibrating function µ applied to it yielding µ(f (x i)). Perfect calibration is defined as the expectation DISPLAYFORM0 where random variables X, Y denote the features and class label of a uniformly randomly drawn instance from the dataset respectively, such that Y = 1, Y = 0 represent an individual positive and negative instance, respectively. To visualize calibration performance of a classifier, BID17 plots the observed frequency of an event against the predicted frequency yielding calibration curves. Calibration curves plot the observed relative frequency against the predicted probability for all test data. Perfect calibration occurs if the calibration curve exactly fits the identity line. Refinement loss measures the difference between a probability estimate and zero or one. This can be combined with a frequency distribution, giving an indication of spread, which is useful if there are few events associated with a particular probability. The notion of refinement is also useful when considering calibration. By considering the crude constant classifier, which predicts the probability corresponding to the class distribution for all inputs, it is clear that this calibration estimate is perfectly calibrated. However, an intuitively more valuable calibration estimate, is one which predicts a value closer to either zero or one. For this reason, BID3 suggests measuring refinement loss, which measures the distance of the classifiers probability estimates to either zero or one. Together, calibration and refinement loss make up the Brier Score. BID10 defines calibration and refinement loss. DISPLAYFORM1 ] is the loss due to the expected difference between the model score S and the proportion of positives among instances (observed relative frequency) with the same score. ] is the loss due to the presence of instances from multiple classes among instances with the same estimate S. In the worst case, this clearly reduces to the crude constant classifier mentioned above. An instance of recent work related to calibration is Beta calibration. Beta calibration BID11 is based on the beta distribution which includes functions such as the logit, sigmoid and identity. This allows it to calibrate scores produced by models such as naive bayes, which biases its scores towards extremities when the assumption of feature independence is not met, using the inverse sigmoid or logit function, a function which is not in softmax's repertoire. In the context of adversarial attacks, the optimization problem that is often formulated to construct adversarial examples is shown. formed example, subject to the constraints above. Intuitively, these constraints coerce the network into mis-classifying each example with a minimal perturbation vector. Example distance metrics that measure the size of the perturbation include L p metrics, as well as the PASS score BID22, a metric designed to better reflect notions of psycho-physical similarity than L p metrics. Adversarial attack methods include L-BFGS from BID23, which uses linear trial and error to find a c for each data instance such that c multiplied by an arbitrary small perturbation vector mis-classifies the instance. FGSM builds on the L-BFGS method, replacing expensive linear search with gradient descent to find the perturbation vector. This uses the following optimization strategy to create the perturbation vector η, such that x is the input to the model, y is the predicted class, θ are the model parameters and J(θ, x, y) is the cost function. x indicates the derivative of the cost function is taken with respect to the input x. The sign function takes the sign of the ing derivative. The adversarial example is then constructed as x = x + η. DISPLAYFORM0 A momentum term can be added to the gradient descent process to yield the work of BID4. The BIM attack by BID12, uses a smaller noise vector produced by FGSM, applied iteratively, before a clip operation is applied to the ing image, keeping it within the maximum image pixel values after each iteration. JSMA is a forward derivative approach, by BID19, which uses a Jacobian matrix, to produce an adversarial saliency map, which indicate the features that when (positively or negatively) perturbed, most efficiently achieve a desired network output. DeepFool BID16 finds an image that is at a minimal distance to the decision boundary from the proposed example, to another target class which isn't the source class, treating multi-class classifiers as combinations of binary affine classifiers. Background Check, introduced by BID21, is the calibrating procedure that our proposed method uses to defend neural networks from adversarial attacks. Background Check takes as input, the scores from logit vectors and maps them to probabilities, replacing softmax after training. Background Check provides a framework to classify regions where no previous data has been seen as regions, that in the context of the proposed method will be classified as regions where adversarial data may lie. Background Check also provides a framework to resolve ambiguity inherent in a single value representing probability. More specifically, Background Check provides two values to represent a single probability value of a data instance. One value represents distance from the data density, and another represents the certainty of a particular class. This approach avoids overloading the meaning of a single number representing probability. More specifically, an uncertainty of 1 /n for all classes, could represent an instance very close to the decision boundary or very far away from training data. On the other hand, an output of zero, could represent a point very far away from training data or a classification that the data is definitely not that particular class. Background Check introduces an additional outlier class, b, representing regions of space where data is sparse or non-existent. Then, a foreground class is introduced, f. This represents regions of space where data is plentiful or dense, i.e. data from any class apart from the class, is abundant. b is introduced as an additional class whilst f is kept as a reference class. Every instance x necessarily belongs to either f or b. P (b|x) = 0 and P (f |x) = 1 refers to absolute certainty that the instance belongs to one of the classes with sufficient training data, where P (C|x) is a conditional probability measure. The ratio of the two conditional measures defines the reliability factor r(x). DISPLAYFORM0 If r(x) ≤ 1, the classification that the reliability factor indicates is b, else if r(x) > 1 then the classification is f. P (x, f) and P (x, b) are referred to as the foreground and densities. Furthermore, the relative foreground and densities can be defined, q f (x) and q b (x). DISPLAYFORM1 Intuitively, the relative density outputs the proportion of f or b, at the point in space corresponding to the instance x being evaluated. Simple dividing q f (x) by q b (x) yields r(x). DISPLAYFORM2 To construct q b (x), four inductive biases, in increasing strength, are given. This work only uses the third inductive bias.1. Inductive bias 1: q b (x) is a function of q f (x). This is justified, by the idea that with no other information, there is no reason to assign different densities to points with the same foreground density. The domain knowledge informs the function used µ: DISPLAYFORM3 2. Inductive bias 2: monotonicity of µ, that is, when moving to a region with higher foreground density the increases or decreases.3. Inductive bias 3: an affine bias, i.e. µ(x) = ax + b or by replacing a and b: DISPLAYFORM4 4. Inductive bias 4: constant ie µ, µ = 0.5. The key notion in Background Check is the implementation of the inductive bias that shapes q b. This implementation is provided in two different different ways.1. BCD: referred to as the discriminative approach, involves generating artificial instances around foreground data and then training a binary discriminative classifier to separate them. The instances are generated in a hypercube or a hypersphere, such that the is half as dense as max x P (x, f).2. BCF: referred to as the familiarity approach, this involves fitting a one class model on the foreground data to obtain q f, then using an inductive bias to obtain q b. The data, x, that is being fit must have an underlying measure. The familiarity factor r(x) can be found, allowing, the posterior probabilities P (b|x) and P (f |x) to be computed. The implementation that the proposed method in this work uses, is the BCF method for its speed in high dimensional spaces. The measure underlying the space was the one-dimensional L 1 difference between elements of the logit vector. For instance, given a two-class score vector [−5, 5], then the score difference is 10. This measure represents the distance of a data point to the softmax decision boundary. The one-class model used to fit q f (x) was a gamma function optimized using maximum likelihood estimation. To establish the link from q f (x) to q b (x), the third inductive bias was used with µ = 1 and µ = 0, with domain knowledge informing the use of a power value. This link manifests itself in the equation below. DISPLAYFORM0 One neural network for each attack type, parameter combination and dataset, is trained with the Adam optimizer . A batch size of 256 and a learning rate of 0.001 is employed. The biases of the neurons are set to 0 and the weights are sampled uniformly at random Figure 1: Background Check applied to the score difference of two, two class neural network logit vectors (one network each for the top and bottom figures). The training, test and adversarial images, in blue, orange and green, respectively, are organised into ten bins each. The adversarial data for the top figure was generated by the momentum attack, with very large perturbations applied. The bottom figure has adversarial data crafted by the JSMA attack with a moderate perturbation vector applied. The green line represents q f, the foreground relative density and q b represents the relative density. It is clear, that the adversarial images in both cases are distinctly separated from the training and test data, which overlap to the point of in-distinguishability. In the top figure, the adversarial images have logit differences much larger than the training/test data compared to much smaller logit differences for the bottom figure. from the interval. Each neural network is tasked with classifying a variety of paired class combinations from the CIFAR10 dataset. These pairs were: dog versus plane (DVP), fish vs ship (FVS) and airplane vs horse (AVH). Regularization techniques such as L 2 regularization and dropout, with p = 0.5 were applied to the networks. The training vs test ratio is 5:1, such that each class has exactly 1000 test images and 5000 training images. The images were pre-processed such that all image pixel values were floating point values between zero and one, rather than values between 0 and 255. Seven different adversarial attacks, from the cleverhans API BID18, were tested against the networks with a variety of parameter combinations. The adversarial images were generated from test data. The attacks were all white-box attacks and performed on the network which included a final softmax layer in its structure. The final two-class average recall of each network on the validation set of the network was always above 80% after only 300 iterations over the training data. The attacks had different effects on each image, for identical parameter settings and as such, the parameters were subjectively chosen based on whether the image fell into one of four classes. Some attacks, due to a mixture of constraints and difficulty in searching the adversarial image space, only had images in one or two of the available classes.1. Large -Image consists entirely of noise.2. Typical -Moderate noise levels, underlying image recognizable.3. Small -Recognizable noise, but clear image.4. Very Small -No noticeable noise, clear image. For instance, BIM an iterative method had 10 iterations, with: 0.8 and i: 0.05, for a large perturbation vector, yet the method from BID15, a non-iterative method had: 12.0. The full parameter settings are listed in the appendix. The average recall, defined in the equation below, rather than accuracy, is evaluated due to the presence of varying class proportions. The 2-class average recall is on the two classes in CIFAR-10. The 3-class average recall includes the adversarial class. DISPLAYFORM0 The table of demonstrates that large perturbation vectors, associate with a mean reduction in average recall of 11.6, whereas for very small perturbation vectors, the mean reduction in average recall is 35.7. Typical perturbation vectors have a mean reduction in average recall of 6.9. In all cases except for two, the average recall decreases. It is clear that the adversarial TPR generally increases as the size of the perturbation vector increases. In particular, three out of the five large perturbation vectors achieve a TPR on the adversarial class of 100%. All models achieve higher average recall than the baseline, which was simply a strategy that with uniform randomness guesses the class. In order to visualize areas where Background Check assigns foreground and densities, it is helpful to construct histograms. When constructing histograms of the test, training and adversarial L 1 logit differences, three categories were established over the space.1. Adversarial examples in a distinct cluster, closer to the decision boundary, than the training and test data.2. Adversarial examples scattered amongst the training/test data.3. Adversarial examples in a distinct cluster, further from the decision boundary, than the training and test data. The JSMA and DeepFool attacks found logit differences smaller than the test and training logits, yet still high enough to yield a significant confidence level when applied to the softmax function. The Madry, Momentum and BIM attacks produced logit differences far higher than the test and training logit differences. However, some attacks found logit differences within the test and training distributions. Figure 2: Madry attack from BID13 in category 3, with a large perturbation vector applied. The differences between the per-class logit values are large, at least 70, albeit less than some of the other attacks which have score differences in the hundreds. The adversarial images with scores represented by the green histogram are very far away from the training and test data, which means that Background Check will separate them well. The central example successful adversarial image is noise, due to the large perturbation applied. The top left confusion matrix is that of the neural network without Background Check. The top right confusion matrix is that of the neural network with Background Check applied to it. The confusion matrices are relatively coloured with the true labels on the vertical axis and the predicted labels on the horizontal axis. The histogram in the bottom of each image uses ten bins, whose size is chosen by scipy, relative to the spread of the data. The y-axis or height of each bar of the blue, orange and green histograms represent the relative frequency of examples in the training, test and adversarial classes, respectively. The x-axis represents the score difference between the logit vectors. The left hand confusion matrix, on the bottom row, shows the attack had equivalent ease generating adversarial images from either class. The right hand confusion matrix shows a reduction in average recall due to the many, 198/1000 false negatives, for the first class, and 136/1000 for the second, which, because the classifier is evaluated on the average recall, will make a large difference to the final average recall. For the histograms of all of the methods, please see the appendix. Figure b) shows the trend of the pairwise differences as the attack parameters increase for the attack from BID13 7 DISCUSSIONBackground check models the data density over the logit differences. It is clear from the figures in the appendix that the adversarial attacks find examples in regions where the test and training data do not exist. Thus, Background Check improves the discriminative performance of the classifier when dealing with adversarial examples with large perturbation vectors. These ing images are noisy and hard to allocate a non-ambiguous class, though we argue that these images can occur and be just as damaging in the real world and as such need to be defended against. It would be useful to follow on from this method with an analysis of generative probability estimation and a corresponding measure of calibration and refinement loss. Promising future research would scale up the logit difference metric underlying Background Check to higher dimensional spaces to deal with a full ten classes to allow for comparability to mainstream literature on adversarial defences. Possible metrics that can underlie Background Check could use the energy distance. In addition, Background Check could be applied to each layer of a neural network. However, this must be setup such that it does not interfere with the ability of the neural network to generalize. The performance of Background Check can be measured in different ways. For example, BID1 proposes performance measures for classification systems with the rejection option. These measures consist of metrics such as the non-rejected accuracy, which measures the ability of the classifier to accurately classify non-rejected samples. The classification quality, which measures the correct decision making of the classifier with the rejector and finally, the rejection quality, which measures the ability to concentrate all mis-classified samples onto the set of rejected samples. A novel approach to defending neural networks against adversarial attacks has been established. This approach intersects two previously unrelated fields of machine learning, calibration and adversarial defences, using the principles underlying Background Check. This work demonstrates that adversarial attacks, produced as a of large perturbations of various forms, can be detected and assigned to an adversarial class. The larger the perturbation, the easier it was for the attacks to be detected.
This paper uses principles from the field of calibration in machine learning on the logits of a neural network to defend against adversarial attacks