On the Security of Machine Learning in Malware C&C Detection: A Survey <s> DNS Traffic Analysis. EXPOSURE <s> Malicious Web sites are a cornerstone of Internet criminal activities. As a result, there has been broad interest in developing systems to prevent the end user from visiting such sites. In this paper, we describe an approach to this problem based on automated URL classification, using statistical methods to discover the tell-tale lexical and host-based properties of malicious Web site URLs. These methods are able to learn highly predictive models by extracting and automatically analyzing tens of thousands of features potentially indicative of suspicious URLs. The resulting classifiers obtain 95-99% accuracy, detecting large numbers of malicious Web sites from their URLs, with only modest false positives. <s> BIB001 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> DNS Traffic Analysis. EXPOSURE <s> In recent years Internet miscreants have been leveraging the DNS to build malicious network infrastructures for malware command and control. In this paper we propose a novel detection system called Kopis for detecting malware-related domain names. Kopis passively monitors DNS traffic at the upper levels of the DNS hierarchy, and is able to accurately detect malware domains by analyzing global DNS query resolution patterns. ::: ::: Compared to previous DNS reputation systems such as Notos [3] and Exposure [4], which rely on monitoring traffic from local recursive DNS servers, Kopis offers a new vantage point and introduces new traffic features specifically chosen to leverage the global visibility obtained by monitoring network traffic at the upper DNS hierarchy. Unlike previous work Kopis enables DNS operators to independently (i.e., without the need of data from other networks) detect malware domains within their authority, so that action can be taken to stop the abuse. Moreover, unlike previous work, Kopis can detect malware domains even when no IP reputation information is available. ::: ::: We developed a proof-of-concept version of Kopis, and experimented with eight months of real-world data. Our experimental results show that Kopis can achieve high detection rates (e.g., 98.4%) and low false positive rates (e.g., 0.3% or 0.5%). In addition Kopis is able to detect new malware domains days or even weeks before they appear in public blacklists and security forums, and allowed us to discover the rise of a previously unknown DDoS botnet based in China. <s> BIB002 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> DNS Traffic Analysis. EXPOSURE <s> Botnets continue to be a significant problem on the Internet. Accordingly, a great deal of research has focused on methods for detecting and mitigating the effects of botnets. Two of the primary factors preventing the development of effective large-scale, wide-area botnet detection systems are seemingly contradictory. On the one hand, technical and administrative restrictions result in a general unavailability of raw network data that would facilitate botnet detection on a large scale. On the other hand, were this data available, real-time processing at that scale would be a formidable challenge. In contrast to raw network data, NetFlow data is widely available. However, NetFlow data imposes several challenges for performing accurate botnet detection. In this paper, we present Disclosure, a large-scale, wide-area botnet detection system that incorporates a combination of novel techniques to overcome the challenges imposed by the use of NetFlow data. In particular, we identify several groups of features that allow Disclosure to reliably distinguish C&C channels from benign traffic using NetFlow records (i.e., flow sizes, client access patterns, and temporal behavior). To reduce Disclosure's false positive rate, we incorporate a number of external reputation scores into our system's detection procedure. Finally, we provide an extensive evaluation of Disclosure over two large, real-world networks. Our evaluation demonstrates that Disclosure is able to perform real-time detection of botnet C&C channels over datasets on the order of billions of flows per day. <s> BIB003
Kopis BIB002 ] makes use of the global view of the upper DNS hierarchy. In Kopis, a classifier is built that, instead of looking at the domains' IP and name, looks at the hosts that make the DNS requests. This leverages the fact that malware-related domains are likely to have an inconsistent, varied pool of requesting hosts compared to a legitimate domain that will be much more consistent. The geographic location is also taken into account: requesters inside large networks are given higher weighting, as a large network is more likely to contain infected machines. A feature set representing this information is used to train a random forest classifier. When tested on 5 months of data taken from two authoritative name servers, the system achieved up to a 98% TP rate with a 0.5% FP rate, and it was even able to identify a new botnet based in China, which was later removed from the Internet. While not explicitly analyzing DNS traffic, BIB001 test the effectiveness of three popular classifiers (Bayes, SVM, and logistic regression (LR)) in identifying malicious domains. Datasets were created by combining blacklist data (for malicious domains) and online directory data (for benign domains) and then producing a dataset using several automatically extracted features. When tested using a 50-50 training/testing split, the SVM and LR algorithms achieved error rates of 1% to 3%. The interesting point is that the Bayes classifier, often chosen for its speed and scalability, achieved a higher error rate of 2.5% to 5%. The authors also tested the case where test data from a different dataset (different directory and blacklist) to the dataset that was used for training is used. The error rate was increased to 44%, which shows that in some cases training and testing on data from the same source may not give accurate real-world results. 4.4.3. Malicious Server Detection. Probing is not the only technique used for detecting servers. BIB003 propose DISCLOSURE, a system for identifying malicious servers from netflow data, extracting features related to flow sizes, client access patterns, and temporal behavior. In particular, the system is designed to work on very large (ISP-level) datasets, for which sampling may have been applied. Labeled netflow records are used to train a random forest classifier, which is able to achieve detection rates of 60% to 70%, with FP rates of 0.5% to 1%. To reduce FPs, information about autonomous systems (ASs) is taken from three public malicious server lists (FIRE, EXPOSURE, and Google Safe Browsing), and a reputation score is calculated for each AS using the information from all three sources. Servers found on networks with an AS score below a threshold are assumed to be FPs and thus are ignored, as the associated network is assumed to not participate in malicious activities. The authors also test the evasion resilience of the system by creating simulated netflow traffic for two botnets that introduce random delays between network connections, and random padding to vary flow lengths, making the botnets appear more like benign servers. By feeding this netflow data into their classifier, they were able to (a) successfully detect the botnets, (b) improve general detection rates, and (c) detect botnets that were not previously detected by their system (the netflow data to which the detection was applied was different from the data input into the classifier).