A Survey on Machine Learning Adversarial Attacks <s> I. INTRODUCTION <s> Deep neural networks and machine-learning algorithms are pervasively used in several applications, ranging from computer vision to computer security. In most of these applications, the learning algorithm has to face intelligent and adaptive attackers who can carefully manipulate data to purposely subvert the learning process. As these algorithms have not been originally designed under such premises, they have been shown to be vulnerable to well-crafted, sophisticated attacks, including training-time poisoning and test-time evasion attacks (also known as adversarial examples). The problem of countering these threats and learning secure classifiers in adversarial settings has thus become the subject of an emerging, relevant research field known as adversarial machine learning. The purposes of this tutorial are: (a) to introduce the fundamentals of adversarial machine learning to the security community; (b) to illustrate the design cycle of a learning-based pattern recognition system for adversarial tasks; (c) to present novel techniques that have been recently proposed to assess performance of pattern classifiers and deep learning algorithms under attack, evaluate their vulnerabilities, and implement defense strategies that make learning algorithms more robust to attacks; and (d) to show some applications of adversarial machine learning to pattern recognition tasks like object recognition in images, biometric identity recognition, spam and malware detection. <s> BIB001 </s> A Survey on Machine Learning Adversarial Attacks <s> I. INTRODUCTION <s> Deep learning is at the heart of the current rise of machine learning and artificial intelligence. In the field of Computer Vision, it has become the workhorse for applications ranging from self-driving cars to surveillance and security. Whereas deep neural networks have demonstrated phenomenal success (often beyond human capabilities) in solving complex problems, recent studies show that they are vulnerable to adversarial attacks in the form of subtle perturbations to inputs that lead a model to predict incorrect outputs. For images, such perturbations are often too small to be perceptible, yet they completely fool the deep learning models. Adversarial attacks pose a serious threat to the success of deep learning in practice. This fact has lead to a large influx of contributions in this direction. This article presents the first comprehensive survey on adversarial attacks on deep learning in Computer Vision. We review the works that design adversarial attacks, analyze the existence of such attacks and propose defenses against them. To emphasize that adversarial attacks are possible in practical conditions, we separately review the contributions that evaluate adversarial attacks in the real-world scenarios. Finally, we draw on the literature to provide a broader outlook of the research direction. <s> BIB002 </s> A Survey on Machine Learning Adversarial Attacks <s> I. INTRODUCTION <s> Deep learning has emerged as a strong and efficient framework that can be applied to a broad spectrum of complex learning problems which were difficult to solve using the traditional machine learning techniques in the past. In the last few years, deep learning has advanced radically in such a way that it can surpass human-level performance on a number of tasks. As a consequence, deep learning is being extensively used in most of the recent day-to-day applications. However, security of deep learning systems are vulnerable to crafted adversarial examples, which may be imperceptible to the human eye, but can lead the model to misclassify the output. In recent times, different types of adversaries based on their threat model leverage these vulnerabilities to compromise a deep learning system where adversaries have high incentives. Hence, it is extremely important to provide robustness to deep learning algorithms against these adversaries. However, there are only a few strong countermeasures which can be used in all types of attack scenarios to design a robust deep learning system. In this paper, we attempt to provide a detailed discussion on different types of adversarial attacks with various threat models and also elaborate the efficiency and challenges of recent countermeasures against them. <s> BIB003 </s> A Survey on Machine Learning Adversarial Attacks <s> I. INTRODUCTION <s> Machine learning is one of the most prevailing techniques in computer science, and it has been widely applied in image processing, natural language processing, pattern recognition, cybersecurity, and other fields. Regardless of successful applications of machine learning algorithms in many scenarios, e.g., facial recognition, malware detection, automatic driving, and intrusion detection, these algorithms and corresponding training data are vulnerable to a variety of security threats, inducing a significant performance decrease. Hence, it is vital to call for further attention regarding security threats and corresponding defensive techniques of machine learning, which motivates a comprehensive survey in this paper. Until now, researchers from academia and industry have found out many security threats against a variety of learning algorithms, including naive Bayes, logistic regression, decision tree, support vector machine (SVM), principle component analysis, clustering, and prevailing deep neural networks. Thus, we revisit existing security threats and give a systematic survey on them from two aspects, the training phase and the testing/inferring phase. After that, we categorize current defensive techniques of machine learning into four groups: security assessment mechanisms, countermeasures in the training phase, those in the testing or inferring phase, data security, and privacy. Finally, we provide five notable trends in the research on security threats and defensive techniques of machine learning, which are worth doing in-depth studies in future. <s> BIB004 </s> A Survey on Machine Learning Adversarial Attacks <s> I. INTRODUCTION <s> Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics. ML is now pervasive—new systems and models are being deployed in every domain imaginable, leading to widespread deployment of software based inference and decision making. There is growing recognition that ML exposes new vulnerabilities in software systems, yet the technical community's understanding of the nature and extent of these vulnerabilities remains limited. We systematize findings on ML security and privacy, focusing on attacks identified on these systems and defenses crafted to date.We articulate a comprehensive threat model for ML, and categorize attacks and defenses within an adversarial framework. Key insights resulting from works both in the ML and security communities are identified and the effectiveness of approaches are related to structural elements of ML algorithms and the data used to train them. In particular, it is apparent that constructing a theoretical understanding of the sensitivity of modern ML algorithms to the data they analyze, a la PAC theory, will foster a science of security and privacy in ML. <s> BIB005
HILE Artificial Intelligence (AI) ethics tends to get the most public attention, there is an increasingly concern about the issue of adversarial attacks to Machine Learning (ML). Attacks to ML products is an emerging problem that has not been addressed by many companies. This survey is ma inly based on Polyakov work  and it tries to provide a structured and comprehensive overview of the research on attacks to ML products. It has been grouped existing techniques into different categories according to the taxonomy recently published by NIST  . For each category, it is identified key assumptions, which are used by the techniques to describe the attack taxonomy. Several papers and posts were reviewed with the aim of identifying those terms and themes, which are the mos t current among authors. There is much overlap between papers and posts, with authors citing the same sources for the topics and terms they discuss. The reader is encouraged to read some works that provided reasonable explanations and compilations reflecting common if not consensus views across a number of authors. Biggio and Roli BIB001 provides a historical study, correlating the evolution of ML with a broader focus on computer vision and cybersecurity tasks. Akhtar and Mian BIB002 , focused on computer vision applications, the deepest address T his work was supported in part by Machine Intelligence and Computer Models Laboratory of Federal University of Rio de Janeiro (IM  on attacks and defenses. Charkraborty et al. BIB003 , Liu et al. BIB004 , and Papernot et al. BIB005 are all concerned with cataloging attacks and defenses with an even broader focus independent of the specific area of application. Pitropakis et al. [8] also present a taxonomy of attacks, but less structured than NIST. The ML key issues of an AI system include data, model, and processes for training, testing, and validation. Although AI also includes various knowledge-based approaches, known as Reasoning Systems, the statistical-driven approach of ML introduces particular security challenges in training and inference phases. The ML methodologies operate with the assumption that their environment is benign, but this assumption does not always hold. These security challenges include the potential for manipulation of training data, and exploitation of model sensitivities to adversely affect the performance of ML classification and regression. Therefore, attacks can occur on two different moments: during training or inference. Attacks during training take place more often than it seems. Most of the production ML systems retrain their prediction models periodically with new data. For instance, social network continuously retrain user's behavior model, which means that each user may interfere in this system by modifying the behavior. Polyakov  organizes the attacks on ML models depending on the actual goal of an attacker (Espionage, Sabotage, Fraud) and the stages of machine learning pipeline (training and inference), or also can be called attacks on algorithm and attacks on a model respectively (see Table 1 ). They are Evasion, Poisoning, Trojaning, Backdooring, Reprogramming, and Privacy attacks. Today, evasion, poisoning and inference are the most widespread. T able 1. Categories of attacks on ML products (Adapt from Polyakov  ).