Generating Realistic Unrestricted Adversarial Inputs using Dual-Objective GAN Training

The correctness of deep neural networks is well-known to be vulnerable to small, 'adversarial' perturbations of their inputs. Although studying these attacks is valuable, they do not necessarily conform to any real-world threat model. This has led to interest in the generation of (and robustness to) unrestricted adversarial inputs, which are not constructed as small perturbations of correctly-classified ground-truth inputs. We introduce a novel algorithm to generate realistic unrestricted adversarial inputs, in the sense that they cannot reliably be distinguished from the training dataset by a human. This is achieved by modifying generative adversarial networks: a generator neural network is trained to construct examples that deceive a fixed target network (so they are adversarial) while also deceiving the usual co-training discriminator network (so they are realistic). Our approach is demonstrated by the generation of unrestricted adversarial inputs for a trained image classifier that is robust to perturbation-based attacks. We find that human judges are unable to identify which image out of ten was generated by our method about 50 percent of the time, providing evidence that they are moderately realistic.


Introduction
In recent years, deep neural networks have achieved impressive results in a wide range of domains. But some of these successes may be precarious, since they often depend on an unwarranted assumption: that the inputs to the trained neural network will always be independently drawn from a fixed probability distribution. This assumption may not be valid because someone deliberately presents inputs they hope will induce incorrect outputs, because the distribution in question changes over time, or simply because the training dataset failed to capture the full distribution of plausibly-occurring inputs.
If this assumption cannot be relied upon, the implications can be dramatic. Szegedy et al. [1] discovered that neural networks are vulnerable to what they termed adversarial inputs: by imperceptibly perturbing correctly-classified inputs in carefully-chosen directions, the accuracy of any state-of-the-art neural network could be almost arbitrarily decreased. Since then, the machine learning community has focused a great deal of research effort on this phenomenon. Many methods for the construction of adversarial perturbations have been introduced [2]. Many attempts to create neural networks resilient to such perturbations have also been suggested [3], along with a variety of approaches aiming to give formal guarantees about the performance of a network when the inputs are adversarial in this sense [4].
Recently, there has been increasing interest in the construction of adversarial inputs that are unrestricted, in the sense that they do not necessarily derive from a perturbation of a natural input [5,6]. Robustness against perturbation-based attacks alone is likely to be necessary but not sufficient for the safe deployment of a neural network if its inputs are not guaranteed to be drawn from a fixed distribution [7].
The main contribution of this paper is a novel and general method for the generation of unrestricted adversarial inputs. The inputs generated by our method also have the important property of being realistic, in the sense that a human cannot reliably distinguish between generated and pre-existing dataset inputs. As described in Section 3, the key idea is to train generative adversarial networks so that the generated examples are not only realistic, but simultaneously are misclassified by some target classifier network.
We demonstrate our method by generating unrestricted adversarial examples of handwritten digits to attack a provably locally-robust classifier trained by Wong and Kolter [8]. Examples are given in Figure 1. In experiments detailed in Section 4, we presented judges with nine dataset images and one generated unrestricted adversarial example, and found that they selected a dataset Figure 1: Ten generated unrestricted adversarial inputs. These were hand-selected as being visually similar to the training dataset, yet the shortest distance between any of these ten and any image in the training dataset is large under all usual l p norms (see Table 1 for details).
image as the most likely to have been generated about 50% of the time, providing evidence of their ability to fool humans. The ability of our method to easily generate unrestricted adversarial examples for a provably locally-robust network provides further confirmation of the need to considering threat models beyond l p -norm constrained perturbations.
This paper is the first step in a broader research aspiration, discussed in Section 6: the automatic generation of tests-in a variety of domains-which are both useful and realistic.

Background
Our method relies on an understanding of generative adversarial networks, explained in this section.

Generative Adversarial Networks
Generative adversarial networks (GANs) [9] are a class of generative machine learning models involving the simultaneous training of two neural networks: a generator g and a discriminator d, with parameters θ and φ respectively. Given a dataset D ⊆ X of samples drawn from a probability distribution p D , the aim is that g implicitly models this distribution by learning to generate samples drawn from it. In particular, the input to g is a random noise variable z ∼ p z , so the goal is that g(z; θ) ∼ p D , where θ are the learned parameters of the generator.
In the original GAN model [9], the output of discriminator network d(x; φ) is intended to represent the probability that a given example x is drawn from the data distribution p D as opposed to being an output of the generator g. The generator and the discriminator are adversarial in that they each attempt to outdo the other during training. The loss function for the generator L g = E z∼pz [− log(d(g(z; θ); φ))] decreases as the discriminator d wrongly predicts that the generated examples are drawn from p D , while the loss function for the discriminator L d = −L g + E x∼p D [− log(d(x; φ))] rewards the discriminator for correctly identifying the source of its inputs.

Wasserstein GANs
While GANs can produce impressive results, their training behaviour is notoriously temperamental. As a result, a large number of modifications to the original GAN architecture have been proposed [10]. The Wasserstein GAN variant [11] aims to make training more stable by designing the discriminator (renamed 'critic') to approximate the Wasserstein distance between the distribution generated by g θ and the data distribution p D , proving a more reliable gradient. Since the dual representation of the Wasserstein distance used in this model requires the function to be 1-Lipschitz continuous, an additional 'gradient penalty' loss term L gp can be added to implement this constraint [12].
The loss functions for this Wasserstein GAN with gradient penalty (WGAN-GP) are: where p I denotes the distribution sampling uniformly at random from the linear interpolations between generated samples and examples from p D .

Conditional GANs
The original proposal for a conditional generative adversarial network (CGAN) learns to generate samples from a conditional distribution [13]; both the generator and the discriminator are given the data y that we wish to condition on in addition to their usual inputs. The training procedure is modified only to appropriately vary y.
An extension of this approach is the auxiliary classifier generative adversarial network (ACGAN) [14], which modifies the structure of the discriminator. Rather than pass the auxiliary information y as an input to the discriminator, the discriminator now gives a secondary output vector predicting the value of y for the input it receives. The loss functions for both the discriminator and generator are given an extra term which decreases when the accuracy of these predictions increases.

Problem Definition
Suppose we have a trained target classifier network f : X → Y which attempts to approximate a function h : X → Y . For most digital domains X, there is only a subset R ⊂ X which may occur naturally. For instance, if X is the set of 4-second audio clips, then only a small subset R of these clips could realistically be an unaltered studio-quality recording of a human voice. Given f and a training dataset D ⊂ R × Y , we aim to generate new test inputsx ∈ X which are: 1. Adversarial in the sense that the neural network's output is incorrect: 2. Realistic in the sense that a human could not reliably identify whether a given input was generated by our method or drawn from the set of naturally-occurring inputs, R.
3. Unrestricted in the sense that the inputx can be in any region of the input space X, unlike perturbationbased approaches, which limit adversarial examples to be close to some known input x ∈ D under some distance metric.  The generator and discriminator are trained by backpropagating loss gradients in the opposite direction to the arrows. This diagram is for high-level intuition only: for specific details, please refer to Algorithm 1. For instance, note that the generator loss is computed using generated inputs only, while the discriminator is trained on a mixture of generated and dataset examples.

Our Method
The key idea is to modify the training of the generator of a generative adversarial network to jointly optimise for two objectives. The first is the usual objective of deceiving the discriminator network so it cannot reliably identify whether a given input was generated by our method or drawn from the dataset D. This incentivises the generator to output realistic inputs. The second objective is the deception of the target network f , incentivising the generation of (unrestricted) adversarial inputs.
The main challenge we overcame was the difficulty in devising a training procedure that successfully optimises both of these objectives. Even assuming that the discriminator gives a good gradient to improve how realistic the generated examples are, simply summing the two loss terms would be unlikely to lead to success. This is because of the relationship between the two loss landscapes: in some cases, it appears that the objectives are locally in conflict. While examples exist that are both realistic and adversarial, the nearest realistic example to a given point may be in the opposite direction from the nearest adversarial example, which is bad news for using a greedy gradient-step algorithm.
For instance, consider a stage in training at which the generated examples would be completely realistic, except for a small consistently-included artefact. Locally, the two objective gradients may be directly opposed: taking a step towards removing the unrealistic artefact may significantly reduce the rate at which the generated input fools the target classifier, yet taking a step towards strengthening the artefact may make it significantly easier for the discriminator to identify the generated examples. As a result, a naive training procedure may get stuck at a local optimum.
There is also reason to believe that the two objectives being locally in conflict is a common occurrence. The phenomenon of adversarial examples was so surprising because it appears that there is an adversarial example very near any point in the input space. Conversely, the set of realistic inputs occupies relatively small and contiguous regions. Therefore, it is likely that a gradient walk from a random point towards a region of realistic inputs is likely to stray dangerously close to regions of unrealistic adversarial inputs, from which the gradient from the loss term incentivising examples to be adversarial may prevent escape.
To minimise the chance of getting stuck in such an unrealistic but adversarial local optimum, we first pretrain the GAN using an ordinary training procedure before adversarially finetuning. By beginning our search for realistic unrestricted adversarial examples in regions of realistic examples, we maximise our chances of converging to nearby optima.

Realistic Pretraining
Any existing GAN algorithm could be used for the pretraining step, allowing our method to leverage the significant advances that are being made in this area. In our experiments, we combine three established architectures: a Wasserstein GAN with gradient penalty (WGAN-GP) [12], a conditional GAN (CGAN) [13] and an auxiliary classifier GAN (ACGAN) [14]. Figure 2 shows the flow of data during training.
The generator g : R m × Y → X is a convolutional neural network. Its inputs are a vector of random noise, which is the source of diversity of its outputs, and a onehot vector specifying the desired true label of the example to be generated.
The discriminator network is a combination of a conditional WGAN-GP critic, which learns an approximation of the Wasserstein distance between the generated and training-set conditional distributions, and an auxiliary classifier, which predicts the likelihood of the possible values of h(x). We combined these two architectures in an attempt to strengthen the gradient provided to the generator, helping to generate data which are both realistic and for which the true (i.e., human-judged) labels match the intended true labels. The critic is given the true label of the data h(x) to improve its training, but the auxiliary classifier must not have access to this information since its purpose is to predict it. We therefore split the discriminator d into three sub-networks. Network d 0 : X → R i effectively preprocesses the input, passing an intermediate representation to the critic network d 1 : R i × Y → R and the auxiliary classifier network d 2 : The loss functions from the WGAN-GP and ACGAN algorithms are combined as might be expected; the training procedure is given as part of Algorithm 1. The loss for the discriminator is the sum of three terms: l d1 rewards the network if d 1 well approximates the Wasserstein distance between the generated and training data; l d2 rewards the network if d 2 accurately classifies the train-ing data; l gp is the WGAN gradient penalty, rewarding the network for being 1-Lipschitz continuous. The loss for the generator is the sum of two terms: l d1 rewards the generator if the d 1 approximation of the Wasserstein distance is low, meaning that the generated images are close to the real distribution; l d2 rewards the network if the training data is given the correct classification by d 2 .
Note that while the values of l d1 and l d2 are not the same for the training of the generator and the discriminator, they appear at the same point in the dataflow and have directly-related interpretations, which is why they share names and block representations in Figure 2.

Adversarial Finetuning
Once we have pretrained a GAN to generate realistic inputs, we perform adversarial finetuning. This is also given in Algorithm 1 and Figure 2. Training continues, but a new loss function is used for the generator network, the product of two terms: l r is the loss used during pretraining, which rewards realistic images; l a is the new term, which rewards the generator if the generated image is misclassified by the target network f . Empirically, we found that taking their product resulted in more realistic images than using their sum. A targeted attack can be used so that l a rewards the network only if the computed label output by f matches a specified target computed label, as opposed to any incorrect label.
Both l r and l a can be negative, yet we need the generator loss to always be a monotonically increasing function of each, so we scale each term using before computing their product. This scaling function has another advantage: the generator does not receive a reward for attaining strongly negative values of either loss term for a given example. This is reminiscent of the clipping parameter in the Carlini & Wagner attack [15], which does not reward high-confidence misclassifications in proportion to their confidence. In our context, this helps to prevents adversarial loss from dominating realistic loss. Note that besides improving the quality of the generated data, our use of a conditional GAN allows us to specify an intended true label to the generator, which the true label of the generated example is very likely to match. In this way, we can reward the generator for generating examples which are very likely to be adversarial by assuming that an example is misclassified if the computed label output by the target classifier does not match the intended true label. Our experiments described in section 4.2 found this assumption to mostly hold.
Trained target neural network, f . Attack rate, µ. 1: procedure TrainGan 2: 6: if odd iteration then 7: if pretraining or t ≥ µ then 12: until convergence Algorithm 2 Compute the loss for the discriminator network given dataset samples x and generated examplesx.
Compute the pretraining and finetuning losses for the generator network given generated examplesx.

Stochastic Loss Selection
While initial experiments gave somewhat promising results, the proportion of generated inputs which were misclassified rose quickly to almost 100%, and stayed there, while the images being generated were not as visually realistic as could be hoped. It seemed that the local optima of the type discussed in section 3.2 were trapping the training process: no steps in the direction of more realistic examples were taken because the gradient was dominated by the loss term l a rewarding misclassifications.
To address this, we introduced a new hyperparameter, the 'attack rate' µ. During the adversarial finetuning stage of training, the finetuning loss function is used only with probability µ; with probability 1−µ, the pretraining loss function is used. These gradient steps can be used to escape local optima of unrealistic adversarial examples.

Experimental Results
Our method aims to generate realistic unrestricted adversarial inputs. We therefore conducted experiments to check whether the generated examples were indeed realistic, unrestricted and adversarial.
The domain we use to evaluate our training process is image classification, making use of the well-known MNIST dataset of handwritten digits [16], since human evaluation of how realistic the generated examples are is particularly easy. The classifier we target is a convolutional network trained by Wong and Kolter [8], which is provably locally robust in the sense that there is guaranteed to be no adversarial input within = 0.1 of 94.2% of test inputs under the l ∞ norm.
We implemented our architecture in Python using the Pytorch library. For each of the ten possible target labels -plus the untargeted case, which aims for any misclassification -a GAN was adversarially finetuned using Algorithm 1. After training converged, each finetuned generator was used to generate examples for all intended true labels. We then filtered these generated images to retain only those inputs for which the computed label matched the target computed label. Images were generated until 200 such filtered examples were generated for each intended true label/target label pairing or until 100 seconds had elapsed. Interestingly, this led to no expected-adversarial examples with intended true label '0' and target classification '1', so this case is omitted. Figure 4 and Appendix F give examples of generated images for which the computed label matches the target classification. Figure 3 gives the same, but attacking a non-robust network.

Does our method generate unrestricted inputs?
Since our method does not work by perturbing existing data, only a simple sanity check was required to verify that the generated images are not close to images in the  [20,8] training set, as could be caused by over-fitting. We selected ten generated adversarial inputs which are visually similar to the training set, and computed the shortest distances between each of the ten and all images in the training set. The selected images are given in Figure 1. Table 1 shows that although the images appear realistic, they are much further from any training example under the usual l p norms than a perturbationbased attack.

Does our method generate unrestricted adversarial inputs?
An unrestricted adversarial input with respect to a target neural network is any input for which the output given by the neural network is incorrect. We hope that the generated inputs for which the label computed by the classifier matches the target label are unrestricted adversarial inputs. However, this is only the case if their true label matches the intended true label given as input to the generator, else it could be that the generator has started simply ignoring this input and generating images which visually match the target class.
To check that the finetuned generators are behaving as hoped, we used workers on Amazon's MTurk platform to label the generated images. We used a sample size of 100 judges for each intended true label/target label pair for each experiment. Figure 5 shows the proportion of inputs for which not only does the label computed by the classifier match the target label, but the human-judged true label matches the intended true label specified to the generator. The mean number of correct labels for the untargeted attack is 80%; this is comparable with the success rate of 85% achieved by Song et al. [6] against a different target classifier. These high rates reassure us that our method really does generate successful unrestricted adversarial inputs. More precisely, the percentage of generated inputs for which the classified label matches the target label which were not identified as being generated when placed amongst nine images from the training dataset. If the generated images were completely realistic, the expected result would be 90.

Are the generated adversarial inputs realistic?
A set of inputs is realistic with respect to a dataset if a human cannot reliably identify to which set an example belongs.
In order to check whether the sets of images generated by the adversarially finetuned GANs are realistic, we again used MTurk workers. After familiarising themselves with examples from the training dataset, each worker was presented with ten images-nine from the dataset and one generated-and asked to identify the image they believed was most likely to be computer-generated rather than a sample from the dataset. If the generated images were entirely realistic, we would expect the generated image to evade selection 9 10 = 90% of the time. If the generated images were entirely unrealistic, we would expect the generated image to be evade identification 0% of the time. Figure 6 shows the proportion of the time that generated adversarial images were identified as such.
The current state of the art, Song et al. [6], perform a similar analysis, except that judges are presented with only two choices each time: one dataset sample and one generated image. For comparability, we also run this experiment for each of the finetuned GANs, with the full results given Appendix E. Targeting Wong and Kolter's locally-robust network [8], Song et al. report that participants do not identify the generated image 21.8% of the time [6]; an untargeted attack using our method against the same network generates images which remain inconspicuous 24% of the time. This difference is probably not statistically significant.

How does performance depend on the target network?
For comparison, we ran the same experiments as above but attacking a non-robust classifier network. Of the generated images for which the computed label matched the intended computed label, the proportion for which the true label also matched the intended true label was 90% (vs. 80% against the robust classifier), and the proportion which were not identified as being generated was 60% (vs. 50% against the robust classifier.) Similar differences were seen in targeted attacks; full results are given in Appendix D, and generated samples are given in Figure 3.

How much does the finetuning step improve performance?
The experiments described above were rerun but using a pretrained rather than finetuned GAN. Unsurprisingly, we found that the desired misclassifications occurred far less frequently than when generated by a finetuned GAN. Furthermore, of the inputs for which the classifier-computed label matched the target classification, the proportion for which the true label also matched the intended true label was also significantly lower without adversarial finetuning: 66% for untargeted attacks and 58% on average for targeted attacks, compared to 80% for both categories after finetuning (see Figure 7). The images from the pretrained GAN for which the computed label matched the target classification were slightly less realistic than those from the finetuned GANs: out of a selection of ten, the generated images were not identified by human judges were 53% and 50% of the time respectively (see Figure 8).
In short, we found that finetuning a GAN using our method roughly maintains how realistic its generated expected-adversarial images are, while significantly increasing their quantity and increasing the proportion which are true adversarial inputs by around 15 percentage points.
Note that while we anecdotally found that by increasing the 'attack rate' hyperparameter, the proportion of images generated which were adversarial could be increased to almost 100%, this was at the cost of making them less realistic; it may be interesting to quantify the nature of this trade-off in the future.

Threats to Validity
We discuss reasons not to believe our claim that our method can generate realistic unrestricted adversarial examples below.

Internal Validity
The evaluation of the generated images relies entirely on the quality of the data provided by the MTurk workers. We therefore took a number of measures to ensure that participants understood the instructions and completed the tasks diligently: • Only workers with good track records were permitted to participate.
• The instructions specified that particular answers should be given to specified questions to prove that the instructions had been read carefully. Approximately 10% of work was rejected for failing this check.
• For the image labelling tasks, some images with known labels were included to check that the right labels were being given. Reassuringly, almost no work was rejected for failing this check.
• For the identification of the generated images, a bonus nearly doubling the pay per image was given for each correctly-identified image, providing an extra incentive to try hard.
• To remove the incentive to save time by randomly clicking answers, a minimum time spent answering each question was enforced.
• If questions were left unanswered, we interpreted this as a sign of carelessness and did not use any of the data from that task.
To best understand the quality and nature of the images generated by our method, we invite you to inspect generated images for yourself in Appendix F.

External Validity
While we can see no reason that our method would fail when applied to other datasets and domains in which GANs have produced good results-indeed, we believe our method will be more successful on more complex datasets, as discussed in Section 5.2-it is true that we have so far only demonstrated it on one simple dataset. We therefore cannot be certain that our method will generalise until we try it.
We look forward to reporting success in applying our adversarial finetuning procedure to a pretrained state-ofthe-art GAN.

Robustness of Neural Networks
Since the discovery that state-of-the-art neural networks are vulnerable to adversarially-crafted perturbations [1], the machine learning community has rightly devoted much effort into developing attacks, defences and provable guarantees centred on local robustness. This is the property that a neural network's performance cannot be significantly degraded by imperceptible perturbations to inputs taken from the training distribution. As we do not have a computable function that is a good proxy for humans' ability to perceive differences, being 'imperceptible' is usually very roughly approximated by constraining perturbations to be with an l p -norm bound (although there have been some successful attacks involving other kinds of meaning-preserving perturbation [21]).
But while there remains agreement that working towards local robustness is a valuable research goal [22], consensus has also grown that safe deployment against any realistic threat model will require something stronger than local robustness [7,23].
Of the plausible threat models suggested by Gilmer et al. [7], the notion of a 'content-constrained input', which has also become known as an 'unrestricted adversarial input', has received the most attention so far, being the focus of a contest [5] and Song et al.'s recent paper [6]. But given the immense challenge that achieving even local robustness poses, we speculate that robustness to unrestricted adversarial inputs is many years away. We suggest that a more achievable research objective might be robustness to realistic unrestricted adversarial examples (or 'non-suspicious' inputs according to Gilmer et al.'s taxonomy [7]).
We expect this to be more achievable since the neural network would be required to behave correctly for a vastly smaller volume of input space. In Brown et al.'s challenge to be robust to unrestricted adversarial inputs [5], any image containing a bicycle should not be classified as a bird. Supposing that input images are 240 by 240 pixels, each of which can take 256 values, there are about 256 200×280 10 135,000 possible images for which the top-left 40 by 40 pixels are a small image of a bike. By contrast, given that naturally-occurring inputs tend to occupy only a low-volume manifold within the input space [24, p. 160], it seems likely that the volume of inputs which could be mistaken for a naturally-occurring input would be much smaller.
A network robust to all realistic unrestricted adversarial inputs could furthermore be safely deployed so long as a human checked that each input appeared plausibly natural; Gilmer et al. describe several scenarios in which this could be applied in practice [7].
Although our method found it around ten percentage points more difficult to generate realistic unrestricted adversarial inputs when the target classifier was locally robust (as described in Section 4.4), it was still able to generate an abundance of such examples. This is clear evidence that local robustness alone does not imply robustness to realistic unrestricted adversarial inputs. Therefore, in ongoing work, we are exploring the extent to which our generated examples can be used to improve robustness to this class of inputs. One of the most successful approaches to improving neural networks' robustness to perturbation-based attacks has been to include adversarial inputs in the training data [25,26,27]; this may prove fruitful in the unrestricted case as well.

Adversarial Input Generation
Song et al. [6] introduce an imaginative method for generating unrestricted adversarial inputs: first train a auxiliary classifier GAN on the dataset, then search in latent space for inputs to the generator that deceive the target network while being confidently correctly classified by the auxiliary classifier. In common with our approach, this method relies on pretraining a GAN to generate realistic examples before using another procedure to obtain unrestricted adversarial examples from these. As discussed in Section 4, the adversarial images generated by both methods are comparable in terms of success rate and realism.
The key difference is the method. Gilmer et al. [7] introduced the idea of a 'test set attack': one source of 'adversarial examples' is simply to try images from test set for which the network gives the wrong answer. Song et al.'s approach can be considered a generalisation of this: assuming that the trained GAN has learnt the data distribution well, their method is simply to search in this distribution for images classified differently by the auxiliary classifier as by the target network. In a sense, this is a search for plausible edge cases within the original distribution. By contrast, our approach does not necessarily search within the original distribution: the loss term rewarding adversarial images pulls away from it, while the loss term from the discriminator pulls towards it, resulting in adversarial examples which are nearby but not necessarily in the original distribution. For this reason, we conjecture that our approach may be more successful at finding unrestricted adversarial inputs for a target network with excellent performance throughout the original data distribution.
Xiao et al. [28] train a GAN to generate adversarial perturbations given a realistic input. The loss function used to train the generator is a linear combination of plain GAN training, the original loss function of the target network (using target labels rather than true labels), and a term regularising the size of the generated perturbations. The crucial difference is that our approach leverages the key advantage of GANs-the generation of realistic examples from scratch-to construct unrestricted adversarial examples, while Xiao et al. use the GAN loss term to ensure that perturbed images remain realistic.
Interestingly, Xiao et al. evaluate whether MTurk workers are able to distinguish adversarially-perturbed images from their originals using a side-by-side test (cf. Appendix E) and find that humans select the perturbed image 50% of the time, no better than random, implying that the adversarial images are entirely realistic. These experiments, however, were performed using high-resolution photographs; the examples of perturbed MNIST images given in the paper have clear visual artefacts that would likely have resulted in a higher identification rate should this dataset have been used instead. We therefore conjecture that our approach to generating realistic unrestricted adversarial examples will in fact be easier with more complex datasets, since any artefacts would be less easy to distinguish in the noisier data. This is consistent with the findings of Shafahi et al. [29], who prove that image classes with lower 'concentration' (i.e., higher within-class variation or complexity) are more susceptible to adversarial perturbations, and observe that this plays out in practice, with MNIST being one of the most difficult datasets to attack.
Both Baluja and Fischer [30] and Hayes and Danezis [31] present different algorithms to train a neural network to generate targeted and 'universal' adversarial perturbations respectively. In common with our approach, these both entail the optimisation of two loss terms: one to ensure the examples are realistic, another to ensure they are adversarial. Of course, the essential difference is that these methods generate small perturbations for existing inputs, while our approach generates the more general unrestricted adversarial examples.
Anderson et al. [32] introduced one of the first approaches to generating unrestricted adversarial examples, using a repurposed autoencoder as a recurrent GAN for the generation of web domain names. But their approach is simply to use ordinary GAN training, meaning that the generated unrestricted examples are only 'adversarial' against the discriminator network of the GAN pair. By contrast, our approach facilitates the generation of unrestricted adversarial examples against any target classifier.

Dual-Objective GAN Training
To the best of our knowledge, relatively little work has been done training GANs to simultaneously optimise for a secondary objective so that the generated data both are realistic and have some other desirable property.
Guimaraes et al. [33] extend the SeqGAN [34] approach of using reinforcement learning to train GANs on discrete sequence by simply adding an additional reinforcement learning reward signal. This approach was applied to generate molecules which were both realistic and 'drug-like', and musical melodies which were both to some extent realistic and tonal.
Cao and Kipf introduced MolGAN [35], which generates drug-like molecules with some success. In addition to the usual GAN loss, the generator was incentivised to maximise a score given to its generated molecules by (a neural network approximation of) a third-party drug-likeness evaluation function.
Besides working with discrete sequential and graphstructured data instead of continuous image data, a significant difference between these works and this paper is that there is unlikely to be local conflict between the two objectives being optimised for: it is not the case that a greedy step in the direction of being more drug-like will be in the opposite direction of being more molecule-like. As discussed in Section 3, overcoming the local conflict between more realistic and more adversarial images is one of the key challenges we address.

Problem: Realistic Test Generation
For many decades, all practical computer systems have realised sufficiently complex input-output functions that certainty in their correctness cannot be found by simply enumerating the entire space of inputs and checking that the system's output on each input is as expected. In response to this problem, the verification community have introduced many successful innovations which provide formal guarantees of system correctness [36,37,38].
But scale remains a huge challenge for formal verification. If formal verification is not a possibility, then testing of some description is the main mechanism by which bugs are found and confidence in a system is increased. Since only a small subset of possible inputs can be used as test cases, the quality of a test suite is far more important than its size. A test suite is high-quality if it is likely to find any bugs which could plausibly be triggered when the system is in deployment.
Given the expense of handwriting test suites, an obvious goal is the automatic generation of test cases. There have been many successes in this field: 'automatic test pattern generation' is very well-established in the hardware domain [39,40]; a large number of test input generation tools have been created for software, from QuickCheck for high-level Haskell programs [41] to bounded model checking for low-level embedded firmware [42] and many others in between [43,44]; there has even been recent work on the generation of tests for neural networks, based on structural coverage metrics [45]. But all approaches to date share a common weakness: the automatically generated tests are not realistic.
Humans leverage their intelligence and their mental model of the system under test to design tests that best cover the space of plausible inputs, system behaviours and system failure modes. In particular, the test inputs crafted are usually realistic, in the sense that they resemble data that the system could plausibly encounter during deployment.
By contrast, the lack of real understanding of a testgeneration program about the system and the environment means that the generated test inputs may technically increase the coverage metric yet not represent the kind of data that would naturally be encountered. Since the aim of testing is to find bugs that could plausibly affect deployment, a method for the generation of realistic test cases would be valuable.

Potential Solution: Dual-Objective-Trained GANs
The original motivation for this work was the generation of realistic test cases. Our research hypothesis is that given a human-written test suite and whitebox access to a system under test, it is possible to train a GAN to generate test inputs that are both realistic and useful in some sense. In this paper, we have shown that it is possible to generate such test inputs for neural networks that are both somewhat realistic and also useful in that they are evidence of a 'bug' relating to the target network's correctness.
In future work, we will explore whether realistic and useful tests can be generated in other contexts. For instance, rather than optimising for adversarial inputs directly, which is only straightforwardly possible when the system under test is differentiable, the finetuning step could optimise to generate tests that maximise some notion of coverage.
The key challenge appears to be coping with non-differentiable systems and metrics; reinforcement learning appears to be a promising solution.

B Neural Network Architectures and Hyperparameters
The WGAN-GP [12] and ACGAN [14] architectures were the starting points for the design of these neural networks. Relatively little hyperparameter tuning was done beyond a little trial and error.

D Results of Experiments for Non-Robust Target Network
These results are targeting a simple convolutional neural network with LeakyReLU activations and three hidden layers of size 256, 128 and 32, trained until convergence.  : Measures of how realistic the adversarial images generated by a pretrained but not finetuned GAN are. More precisely, the percentage of generated inputs for which the classified label matches the target label which were not identified as being generated when placed amongst nine images from the training dataset. If the generated images were completely realistic, the expected result would be 90.