On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Hybrid Detection Systems <s> Botnets are now the key platform for many Internet attacks, such as spam, distributed denial-of-service (DDoS), identity theft, and phishing. Most of the current botnet detection approaches work only on specific botnet command and control (C&C) protocols (e.g., IRC) and structures (e.g., centralized), and can become ineffective as botnets change their C&C techniques. In this paper, we present a general detection framework that is independent of botnet C&C protocol and structure, and requires no a priori knowledge of botnets (such as captured bot binaries and hence the botnet signatures, and C&C server names/addresses). We start from the definition and essential properties of botnets. We define a botnet as a coordinated group of malware instances that are controlled via C&C communication channels. The essential properties of a botnet are that the bots communicate with some C&C servers/peers, perform malicious activities, and do so in a similar or correlated way. Accordingly, our detection framework clusters similar communication traffic and similar malicious traffic, and performs cross cluster correlation to identify the hosts that share both similar communication patterns and similar malicious activity patterns. These hosts are thus bots in the monitored network. We have implemented our BotMiner prototype system and evaluated it using many real network traces. The results show that it can detect real-world botnets (IRC-based, HTTP-based, and P2P botnets including Nugache and Storm worm), and has a very low false positive rate. <s> BIB001 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Hybrid Detection Systems <s> Many botnet detection systems employ a blacklist of known command and control (C&C) domains to detect bots and block their traffic. Similar to signature-based virus detection, such a botnet detection approach is static because the blacklist is updated only after running an external (and often manual) process of domain discovery. As a response, botmasters have begun employing domain generation algorithms (DGAs) to dynamically produce a large number of random domain names and select a small subset for actual C&C use. That is, a C&C domain is randomly generated and used for a very short period of time, thus rendering detection approaches that rely on static domain lists ineffective. Naturally, if we know how a domain generation algorithm works, we can generate the domains ahead of time and still identify and block bot-net C&C traffic. The existing solutions are largely based on reverse engineering of the bot malware executables, which is not always feasible. ::: ::: In this paper we present a new technique to detect randomly generated domains without reversing. Our insight is that most of the DGA-generated (random) domains that a bot queries would result in Non-Existent Domain (NXDomain) responses, and that bots from the same bot-net (with the same DGA algorithm) would generate similar NXDomain traffic. Our approach uses a combination of clustering and classification algorithms. The clustering algorithm clusters domains based on the similarity in the make-ups of domain names as well as the groups of machines that queried these domains. The classification algorithm is used to assign the generated clusters to models of known DGAs. If a cluster cannot be assigned to a known model, then a new model is produced, indicating a new DGA variant or family. We implemented a prototype system and evaluated it on real-world DNS traffic obtained from large ISPs in North America. We report the discovery of twelve DGAs. Half of them are variants of known (botnet) DGAs, and the other half are brand new DGAs that have never been reported before. <s> BIB002
BotMiner BIB001 ] is a system for detecting infected hosts without previous knowledge of specific botnets. In this system, bots are identified by clustering hosts that exhibit similar communication and (suspected) malicious activities. Activities are monitored using custom signatures for the Snort IDS. The clustering (x-means) groups hosts according to the botnet to which they belong, using the fact that hosts within the same botnet are likely to exhibit similar communication patterns and will usually perform activities synchronously (e.g., DDoS attacks). When tested on 10 days of university traffic, with the traffic of eight botnets inserted, the system achieved a 100% TP rate for six out of the eight botnets, with the other two achieving TP rates of 99% and 75%. The system achieves a low FP rate. BIB002 describe a system for identifying previously unseen DGAs by taking advantage of the fact that DGAs result in large amounts of nonexistent domain (NXDomain) responses, and bots within the same botnet will generate similar NXDomain traffic. The system has a three-step process. First, domains are clustered in two different ways using the x-means algorithm, first by string-based features of the domain name and second by domains that share source hosts. These two sets of clusters are then combined through intersection, and any clusters small enough are discarded. These clusters are then passed to a filtering step, which makes use of a multiclass variant of the alternating decision trees (ADT) classifier (trained on previously discovered DGAs) to remove known DGAs. The system then produces a hidden Markov model (HMM) for each DGA that can be used to evaluate single domain names. When tested on the traffic data from a U.S. ISP over a 15-month period, 360,700 NXDomains were discovered, queried by 187,600 distinct hosts. The system identified 12 DGAs, of which 6 were new. For assigning domains to known DGA algorithms using the HMMs, TP rates of 99.7% and FP rates of 0.1% can be achieved.