On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Poisoning Attacks: Classifiers <s> In adversarial classication tasks like spam ltering and intrusion detection, malicious adversaries may manipulate data to thwart the outcome of an automatic analysis. Thus, besides achieving good classication performances, machine learning algorithms have to be robust against adversarial data manipulation to successfully operate in these tasks. While support vector machines (SVMs) have shown to be a very successful approach in classication problems, their eectiveness in adversarial classication tasks has not been extensively investigated yet. In this paper we present a preliminary investigation of the robustness of SVMs against adversarial data manipulation. In particular, we assume that the adversary has control over some training data, and aims to subvert the SVM learning process. Within this assumption, we show that this is indeed possible, and propose a strategy to improve the robustness of SVMs to training data manipulation based on a simple kernel matrix correction. <s> BIB001 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Poisoning Attacks: Classifiers <s> To develop a robust classification algorithm in the adversarial setting, it is important to understand the adversary's strategy. We address the problem of label flips attack where an adversary contaminates the training set through flipping labels. By analyzing the objective of the adversary, we formulate an optimization framework for finding the label flips that maximize the classification error. An algorithm for attacking support vector machines is derived. Experiments demonstrate that the accuracy of classifiers is significantly degraded under the attack. <s> BIB002 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Poisoning Attacks: Classifiers <s> We investigate a family of poisoning attacks against Support Vector Machines (SVM). Such attacks inject specially crafted training data that increases the SVM's test error. Central to the motivation for these attacks is the fact that most learning algorithms assume that their training data comes from a natural or well-behaved distribution. However, this assumption does not generally hold in security-sensitive settings. As we demonstrate, an intelligent adversary can, to some extent, predict the change of the SVM's decision function due to malicious input and use this ability to construct malicious data. ::: ::: The proposed attack uses a gradient ascent strategy in which the gradient is computed based on properties of the SVM's optimal solution. This method can be kernelized and enables the attack to be constructed in the input space even for non-linear kernels. We experimentally demonstrate that our gradient ascent procedure reliably identifies good local maxima of the non-convex validation error surface, which significantly increases the classifier's test error. <s> BIB003
6.3.1. Label-Flipping Attacks. Label-flipping attacks are a form of poisoning attack with a goal of introducing label noise into the training data by flipping labels. The attacker is able to cause an amount of the legitimate samples to be labeled as malicious in the training data or an amount of the malicious samples to be labeled as legitimate. The amount is down to the capability of the attacker. Label flipping is an example of a causative integrity/availability attack that attempts to maximize classification error. There have been many attempts at designing ML algorithms to provide robustness against random label noise arising from noisy datasets, but these designs do not consider the adversarial case where the attacker attempts to maximize the classification error. The extent to which labels can be flipped is restricted by a budget (the number of labels that can be flipped). In the real world, the attacker would need to have an impact on the data collection phase. There are several ways that they could do this. One of the most common methods for collecting malware data is through the use of honeypots and sandboxes. Malware has been shown to be able to identify when it is being run in a virtual environment (see Section 6.6), so the malware could be programmed to engage only in legitimate behavior when in a sandbox, which would then create legitimate samples that are labeled as malicious. Affected algorithms. The attack has been demonstrated against the SVM classifier, although it should affect any classification algorithms affected by label noise. Demonstrated attacks. BIB002 discuss a causative attack against SVMs that makes use of label flipping to poison the training set. In this attack, the adversary flips labels of training points, constrained by a budget, with attack points chosen to inflict the maximum loss. On an artificial, two-dimensional dataset, the attack is able to increase the error rate up to 32% (on a binary class dataset with a 50/50 split), which is not far from reducing the classifier to a random guess. When tested on a set of 10 real-world datasets (downloaded from the LIBSVM Web site), the attack is able to reduce the classifier to a random guess (a 50% error rate) by flipping 10% of the labels of the training set. An interesting observation is that the attack is tested on an SVM classifier trained with both a linear kernel and an RBF kernel, and is found to be much more susceptible to attack when using the RBF kernel. The authors speculate that this is due to the fact that when using the RBF kernel, instances are mapped to the infinite dimensional feature space, and thus instances are more sparsely distributed. Therefore, flipping a label has a greater effect on the hyperplane. This indicates that even variations of the same ML algorithm have different tolerances to the same attack. The attack is also effective to a lesser degree against the label noise robust SVM (LN-SVM) algorithm BIB001 ]. Attack limitations. The primary limitation of the attack is the budget limiting the number of points in the training set that can have their labels flipped. The more restrictive the budget, the more restricted is the attack. The attack also requires some knowledge on behalf of the attacker of the classifier and training data in use. If this information is limited, then the attack becomes far more difficult for the attacker. 6.3.2. Gradient Descent Attacks. Gradient descent-based poisoning attacks impact the availability of the learner. These are causative attacks, which can be either targeted or indiscriminate in nature. As opposed to the evasion case (see Sections 6.1.2 and 6.2.2), where attack points are moved to be misclassified, in the poisoning case, the gradient descent attack inserts attack points into the training data chosen to maximize the impact on the classifier performance, with a goal of reducing performance to the level that the classifier is unusable. The common approach is to start with a benign point, flip its label, and move it (according to the gradient descent function) to maximize the objective function. Affected algorithms. The gradient descent-based poisoning attack has been demonstrated against the SVM. Poisoning attacks work by effectively introducing adversarial noise into the training data. In that sense, a gradient descent-based attack is theoretically effective against any classification algorithm that is not tolerant to noise. Even if an algorithm is tolerant to random noise, it may be possible to apply the attack in a controlled manner and still successfully carry out the attack, although the complexity will be increased. Demonstrated attacks. BIB003 demonstrate a poisoning attack against SVM. The attack assumes knowledge of the training set used by the learner. The attacker inserts attack points by flipping the label of a point in the target class to create an initial attack point and then applies a gradient descent to find an optimal attack point that is then added to the training set. When evaluated on an artificial, two-dimensional dataset, the attack can achieve a classification error of up to 0.06 for a linear kernel and 0.035 for the RBF kernel. The attack is also tested on the handwritten digits dataset mentioned previously using an SVM with the linear kernel, which achieves a classification error of 0.1 to 0.3 after 200 iterations of the gradient descent algorithm. Attack limitations. The attack is limited by the amount of noise that the attacker can inject. One example is the attacker's ability to cause data points to be labeled as they require. For example, when attacking a spam filter, the spam label usually arises from the user, and thus the attacker is reliant on the oracle labeling the points in the way the attacker desires. 6.3.3. Dictionary Attacks. A dictionary attack is a specific form of poisoning attack that can be carried out against classifiers trained using token-based features. In this attack, malicious data is inserted into the training set that contains tokens (e.g., words) which can be found in benign data. The intention is that the malicious data will be discovered and included in future training sets. The goal is then to cause benign data to be misclassified as malicious, indicating a causative availability attack. This can both be indiscriminate (any benign point can be misclassified) or targeted (the attacks want to cause a particular benign sample to be misclassified). provide attacks against SpamBayes, 1 a system for identifying spam email messages. Two causative availability attacks are proposed against the system. The first, an indiscriminate dictionary attack, sends emails to a destination covered by SpamBayes that contains words likely to appear in legitimate emails. These spam emails will be included in the training data when the system is retrained (which happens periodically), increasing the likelihood that legitimate emails will be labeled as spam. In the optimal case, the emails will contain the full English dictionary. As a refinement, the email can just contain a dictionary closer to the victim's word distribution. The second attack, a targeted attack, assumes knowledge of a specific email that the attacker does not want the intended recipient to read by causing it to appear in the victim's spam folder. This attack involves including words specific to that email in the attack emails, and hence the attack effectiveness is limited by the attacker's knowledge of the structure of the target email. In the optimal case, this will be the full dictionary attack, which will contain all of the target words. When tested on a large dataset, the attacks are proved to be effective. The dictionary attack, when attack emails make up 1% of the dataset (achievable by a large-scale attacker), can cause 90% of the legitimate emails to be labeled as unsure or spam. The targeted attack can, with 30% knowledge of the target email, change classification in 60% of cases. The SpamBayes algorithm is shared by Bogofilter 2 and SpamAssassin, 3 and thus the proposed attacks should have a large impact on these systems as well.