A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> In this paper, we provide a survey of the traffic sign detection literature, detailing detection systems for traffic sign recognition (TSR) for driver assistance. We separately describe the contributions of recent works to the various stages inherent in traffic sign detection: segmentation, feature extraction, and final sign detection. While TSR is a well-established research area, we highlight open research issues in the literature, including a dearth of use of publicly available image databases and the over-representation of European traffic signs. Furthermore, we discuss future directions of TSR research, including the integration of context and localization. We also introduce a new public database containing U.S. traffic signs. <s> BIB001 </s> A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> Traffic signs are characterized by a wide variability in their visual appearance in real-world environments. For example, changes of illumination, varying weather conditions and partial occlusions impact the perception of road signs. In practice, a large number of different sign classes needs to be recognized with very high accuracy. Traffic signs have been designed to be easily readable for humans, who perform very well at this task. For computer systems, however, classifying traffic signs still seems to pose a challenging pattern recognition problem. Both image processing and machine learning algorithms are continuously refined to improve on this task. But little systematic comparison of such systems exist. What is the status quo? Do today's algorithms reach human performance? For assessing the performance of state-of-the-art machine learning algorithms, we present a publicly available traffic sign dataset with more than 50,000 images of German road signs in 43 classes. The data was considered in the second stage of the German Traffic Sign Recognition Benchmark held at IJCNN 2011. The results of this competition are reported and the best-performing algorithms are briefly described. Convolutional neural networks (CNNs) showed particularly high classification accuracies in the competition. We measured the performance of human subjects on the same data-and the CNNs outperformed the human test persons. <s> BIB002 </s> A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> Machine-learning (ML) algorithms are increasingly utilized in privacy-sensitive applications such as predicting lifestyle choices, making medical diagnoses, and facial recognition. In a model inversion attack, recently introduced in a case study of linear classifiers in personalized medicine by Fredrikson et al., adversarial access to an ML model is abused to learn sensitive genomic information about individuals. Whether model inversion attacks apply to settings outside theirs, however, is unknown. We develop a new class of model inversion attack that exploits confidence values revealed along with predictions. Our new attacks are applicable in a variety of settings, and we explore two in depth: decision trees for lifestyle surveys as used on machine-learning-as-a-service systems and neural networks for facial recognition. In both cases confidence values are revealed to those with the ability to make prediction queries to models. We experimentally show attacks that are able to estimate whether a respondent in a lifestyle survey admitted to cheating on their significant other and, in the other context, show how to recover recognizable images of people's faces given only their name and access to the ML model. We also initiate experimental exploration of natural countermeasures, investigating a privacy-aware decision tree training algorithm that is a simple variant of CART learning, as well as revealing only rounded confidence values. The lesson that emerges is that one can avoid these kinds of MI attacks with negligible degradation to utility. <s> BIB003 </s> A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> Machine learning is enabling a myriad innovations, including new algorithms for cancer diagnosis and self-driving cars. The broad use of machine learning makes it important to understand the extent to which machine-learning algorithms are subject to attack, particularly when used in applications where physical security or safety is at risk. In this paper, we focus on facial biometric systems, which are widely used in surveillance and access control. We define and investigate a novel class of attacks: attacks that are physically realizable and inconspicuous, and allow an attacker to evade recognition or impersonate another individual. We develop a systematic method to automatically generate such attacks, which are realized through printing a pair of eyeglass frames. When worn by the attacker whose image is supplied to a state-of-the-art face-recognition algorithm, the eyeglasses allow her to evade being recognized or to impersonate another individual. Our investigation focuses on white-box face-recognition systems, but we also demonstrate how similar techniques can be used in black-box scenarios, as well as to avoid face detection. <s> BIB004 </s> A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> We quantitatively investigate how machine learning models leak information about the individual data records on which they were trained. We focus on the basic membership inference attack: given a data record and black-box access to a model, determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on. We empirically evaluate our inference techniques on classification models trained by commercial "machine learning as a service" providers such as Google and Amazon. Using realistic datasets and classification tasks, including a hospital discharge dataset whose membership is sensitive from the privacy perspective, we show that these models can be vulnerable to membership inference attacks. We then investigate the factors that influence this leakage and evaluate mitigation strategies. <s> BIB005 </s> A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> Deep neural networks are susceptible to \emph{adversarial} attacks. In computer vision, well-crafted perturbations to images can cause neural networks to make mistakes such as confusing a cat with a computer. Previous adversarial attacks have been designed to degrade performance of models or cause machine learning models to produce specific outputs chosen ahead of time by the attacker. We introduce attacks that instead {\em reprogram} the target model to perform a task chosen by the attacker---without the attacker needing to specify or compute the desired output for each test-time input. This attack finds a single adversarial perturbation, that can be added to all test-time inputs to a machine learning model in order to cause the model to perform a task chosen by the adversary---even if the model was not trained to do this task. These perturbations can thus be considered a program for the new task. We demonstrate adversarial reprogramming on six ImageNet classification models, repurposing these models to perform a counting task, as well as classification tasks: classification of MNIST and CIFAR-10 examples presented as inputs to the ImageNet model. <s> BIB006 </s> A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> Recent studies show that the state-of-the-art deep neural networks (DNNs) are vulnerable to adversarial examples, resulting from small-magnitude perturbations added to the input. Given that that emerging physical systems are using DNNs in safety-critical situations, adversarial examples could mislead these systems and cause dangerous situations. Therefore, understanding adversarial examples in the physical world is an important step towards developing resilient learning algorithms. We propose a general attack algorithm, Robust Physical Perturbations (RP2), to generate robust visual adversarial perturbations under different physical conditions. Using the real-world case of road sign classification, we show that adversarial examples generated using RP2 achieve high targeted misclassification rates against standard-architecture road sign classifiers in the physical world under various environmental conditions, including viewpoints. Due to the current lack of a standardized testing method, we propose a two-stage evaluation methodology for robust physical adversarial examples consisting of lab and field tests. Using this methodology, we evaluate the efficacy of physical adversarial manipulations on real objects. With a perturbation in the form of only black and white stickers, we attack a real stop sign, causing targeted misclassification in 100% of the images obtained in lab settings, and in 84.8% of the captured video frames obtained on a moving vehicle (field test) for the target classifier. <s> BIB007 </s> A Survey on Machine Learning Adversarial Attacks <s> IV. TROJANING <s> Adversarial attacks on machine learning models have seen increasing interest in the past years. By making only subtle changes to the input of a convolutional neural network, the output of the network can be swayed to output a completely different result. The first attacks did this by changing pixel values of an input image slightly to fool a classifier to output the wrong class. Other approaches have tried to learn "patches" that can be applied to an object to fool detectors and classifiers. Some of these approaches have also shown that these attacks are feasible in the real-world, i.e. by modifying an object and filming it with a video camera. However, all of these approaches target classes that contain almost no intra-class variety (e.g. stop signs). The known structure of the object is then used to generate an adversarial patch on top of it. ::: In this paper, we present an approach to generate adversarial patches to targets with lots of intra-class variety, namely persons. The goal is to generate a patch that is able successfully hide a person from a person detector. An attack that could for instance be used maliciously to circumvent surveillance systems, intruders can sneak around undetected by holding a small cardboard plate in front of their body aimed towards the surveillance camera. ::: From our results we can see that our system is able significantly lower the accuracy of a person detector. Our approach also functions well in real-life scenarios where the patch is filmed by a camera. To the best of our knowledge we are the first to attempt this kind of attack on targets with a high level of intra-class variety like persons. <s> BIB008
OLYAKOV  highlights that in poisoning, attackers don't have access to the model and initial dataset, they only can add new data to the existing dataset or modify it. However, in Trojaning an attacker still do not have access to the initial dataset but have access to the model and its parameters and can retrain this model. This may happen in transfer learning. Most companies do not build their own models from scratch but retrain the existing models. For example, if it is necessary to create a model for workers detection at an industrial scenario, a software house may take the latest image recognition model of person and retrain it with dataset containing people dressing industrial coveralls. This means that most AI companies download popular models from the Internet where hackers can replace them with their own modified versions. Liu et al. [13] describe the method for perform trojaning in ML systems. They inverse the neural network to generate a general trojan trigger, and then retrain the model with reversed engineered training data to inject malicious behaviors to the model. The malicious behaviors are only activated by inputs stamped with the trojan trigger. A trojan trigger is some special input that triggers the trojaned neural network to misbehave. Such input is usually just a small part of the entire input to the neural network (e.g., a logo or a small segment of audio). Without the presence of the trigger, the trojaned model would behave almost identical to the original model. The attacker starts by choosing a trigger mask, which is a subset of the input variables that are us ed to inject the trigger (see Figure 3a ). Then, derive a set of data that can be used to retrain the model in a way that it performs normally when images of the persons in the original training set are provided and emits the masquerade output when the trojan trigger is present ( Figure 3b ). Specifically, it start with an image generated by averaging all the fact images from an irrelevant public dataset, from which the model generates a very low classification confidence (i.e., 0.1) for the target output. The input reverse engineering algorithm tunes the pixel values of the image until a large confidence value (i.e., 1.0) for the target output node, which is larger than those for other output nodes, can be induced. Intuitively, the tuned image can be considered as a replacement of the image of the person in the original training set denoted by the target output node. Moreover, repeat this process for each output node to acquire a complete training set. Finally, use the trigger and the reverse engineered images to retrain part of the model, namely, the layers in between the residence layer of the selected neurons and the output layer ( Figure 3c ). The essence of the retraining I P is to establish the strong link between the selected neurons (that can be excited by the trigger) and the output node denoting the masquerade target, the weight between the selected neuron and the masquerade target node. It also reduces other weights in the neural network, especially those correlated to the masquerade target node to compensate the inflated weights. V. REPROGRAMMING SUALLY adversarial attacks are untargeted attacks that aim to compromise the performance of a model without necessarily requiring it to produce a specific output. This is quite different from targeted attacks in which the attacker designs an adversarial perturbation to produce a specific output for that input. For example, an attack against a classifier might target a specific desired output class for each input image, or an attack against a reinforcement learning agent might induce that agent to enter a specific state [14] . Elsayed et al. BIB006 consider a novel and more challenging adversarial goal: reprogramming the model to perform a task chosen by the attacker, without the attacker needing to compute the specific desired output. Consider a model trained to perform some original task: for inputs x it produces outputs f(x). Then, consider an adversary who wishes to perform an adversarial task: for inputs y (not necessarily in the same domain as x) the adversary wishes to compute a function g(y). The authors demonstrate adversarial programs that target several convolutional neural networks designed to classify ImageNet data. These adversarial programs alter the network function from ImageNet classification to: counting squares in an image, classifying MNIST digits, and classifying CIFAR-10 images. Adversarial attacks allowed them to create images that resembled a specific noise containing several small white squares inside a big black square. They chose the pictures in the way that, for example, the network considered the noise with a white square on a black background to be a tench, and the noise with two white squares to be a goldfish, etc (see Figure 4 ). The image recognition system became a model that can calculate the number of squares in the picture. In a broader perspective, says Polyakov  , attackers can use some open ML Application Programming Interface (API) for image recognition to solve other tasks that they need, and use the resources of the target ML model. Fig. 4 . Evasion attack against a deep neural network prediction model BIB006 . Figure 5 , is also taken from Elsayed et al. BIB006 illustrate different adversarial programs targeted to repurpose networks pre-trained on ImageNet to count squares in images, to function as MNIST classifiers, and to function as CIFAR-10 classifiers. VI. PRIVACY ATTACK RIVACY attacks intend to explore the system, such as model, or dataset that can further be useful. In this survey, three types of privacy attacks are presented: membership inference BIB005 , model inversion BIB003 , model extraction [18] . The membership inference is one of the immediate attack against ML systems. It quantitatively investigate how ML models leak information about the individual data records on which they were trained. Given a data record and black-box access to a model, one wants to determine if the record was in the model's training dataset. To perform membership inference against a target model, we make adversarial use of machine learning and train our own inference model to U P recognize differences in the target model's predictions on the inputs that it trained on versus the inputs that it did not train on BIB005 . Membership inference may be used as an exploratory phase for Evasion attacks. The model inversion, also called input inference, is a common attack type. Unlike membership inference where someone wants to guess whether an example was in the training dataset, here someone wants to actually extract data from the training dataset. While dealing with images, it's possible to extract a certain image from the dataset, for instance, just knowing the name of a person, you can get his or her photo. In terms of privacy, this presents a big issue for any system, especially today when General Data Protection Regulation (GDPR) compliance is a hotspot. Fredrikson et al. BIB003 describe model inversion attack against face recognition ML systems, where the attacker is given only the person's name and access to a facial recognition system that returns a class confidence score. These ML models are quickly becoming the standard by which facial recognition systems are evaluated, so the authors consider three types of neural network models: softmax regression, a multilayer perceptron network (MLP), and a stacked denoising autoencoder network (DAE). These models vary in complexity, with softmax regression being the simplest and the DAE being the most complex. Figure 6 show an image recovered using a the model inversion attack. The model extraction, also called parameter inference, is the less common attack. The goal of this attack is to know the exact model or even a model's hyperparameters. This information can be useful for attacks like Evasion in the blackbox environment. Figure 7 shows a data owner that has a model f trained on its data and allows others to make prediction queries. An adversary uses q prediction queries to extract an f ˆ ≈ f [18] . Fig. 7 . Diagram of ML model extraction attacks [18] . VII. ATTACKS IN T HE PHYSICAL WORLD F course the attacks described so far in this paper are also being conducted in the physical world. Some attacks are ordinary use of techniques described so far in this article. Others are solutions to corrupt the image acquisition systems which will provide bad images to the ML systems. The increasingly vast suite of surveillance tools available to state authorities has certainly given privacy advocates something to bristle at. In an exhibition, the artist Adam Harvey (see Figure 8 ) and fashion the designer Johanna Bloomfield demonstrated fashion's potential to thwart surveillance by state actors via accessories like a heat-cloaking anti-drone hoodie and scarf  , and a series of blocky images that could become the building blocks of anti-surveillance makeup  . Fig. 8 . Heat-cloaking wearbles (adapted from Dillow  ) and antisurveillance makeup  . Xu et al. [21] propose what they called an "adversarial" Tshirt, one with a printed image that evades person-detectors even when it is deformed by a wearer is changing pose. They claim it manages to achieve up to 74% and 57% success rates in digital and physical worlds, respectively, against the popular YOLOv2 model (see Figure 9 ). Thys et al. BIB008 have implemented a similar approach. They show how simple printed patterns can fool an AI system that was designed to recognize people in images (YOLOv2). If you print off one of the students' specially designed patches and hang it around your neck, from an AI's point of view, you may as well have slipped under an invisibility cloak. Yamada et al. [23] has developed eyeglasses that help users protect their privacy by disabling facial-recognition systems in cameras. They prototype made two types of glasses, one using O near-infrared light and other using reflectors to fool the cameras into not seeing a face (see Figure 10 ). Moreover, it seems not necessary to create expensive eyeglasses to compromise ML systems. Sharif et al. BIB004 have shown that specially designed spectacle frames can fool even state-of-the-art facial recognition software. Not only can the glasses make the wearer essentially disappear to such automated systems, it can even trick them into thinking you are someone else. By tweaking the patterns printed on the glasses, the authors were able to assume one another's identities or make the software think they were looking at celebrities (see Figure 11 ). An expensive solution is the video of a black mirror-esque wearable face projector capable of tricking facial recognition systems, created by art and product designer Jing-Cai Lu  . The false faces being projected onto the individual can be seen shifting left and right, despite the wearer's head being still, indicating that the light could be coming from in front of them. Given that the Figure 12 snapshot is being filmed at night, it remains to be seen whether such an item would be usable during the daytime. Eykholt et al. BIB007 proposed a white-box adversarial sample generation method to attack their own trained road sign recognition models, including LISA-CNN models used LISA BIB001 , a U.S. traffic sign dataset containing 47 different road signs, and GTSRB-CNN models, which trained on the German Traffic Sign Recognition Benchmark (GTSRB) BIB002 . They proposed two effective kinds of disturbance installation methods for road sign recognition scenarios, i.e., posters and stickers, as shown in Figure 13 . They followed Sharif et al. BIB004 in constructing the loss function and took into account the printability and location limitations. Their assessment showed that they had achieved a 100% success rate in the poster installation driving test. 