On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Clustering-Based Methods <s> Stealthy malware, such as botnets and spyware, are hard to detect because their activities are subtle and do not disrupt the network, in contrast to DoS attacks and aggressive worms. Stealthy malware, however, does communicate to exfiltrate data to the attacker, to receive the attacker's commands, or to carry out those commands. Moreover, since malware rarely infiltrates only a single host in a large enterprise, these communications should emerge from multiple hosts within coarse temporal proximity to one another. In this paper, we describe a system called Ti¾?md (pronounced "tamed") with which an enterprise can identify candidate groups of infected computers within its network. Ti¾?md accomplishes this by finding new communication "aggregates" involving multiple internal hosts, i.e., communication flows that share common characteristics. We describe characteristics for defining aggregates--including flows that communicate with the same external network, that share similar payload, and/or that involve internal hosts with similar software platforms--and justify their use in finding infected hosts. We also detail efficient algorithms employed by Ti¾?md for identifying such aggregates, and demonstrate a particular configuration of Ti¾?md that identifies new infections for multiple bot and spyware examples, within traces of traffic recorded at the edge of a university network. This is achieved even when the number of infected hosts comprise only about 0.0097% of all internal hosts in the network. <s> BIB001 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Clustering-Based Methods <s> The Domain Name System (DNS) is an essential protocol used by both legitimate Internet applications and cyber attacks. For example, botnets rely on DNS to support agile command and control infrastructures. An effective way to disrupt these attacks is to place malicious domains on a "blocklist" (or "blacklist") or to add a filtering rule in a firewall or network intrusion detection system. To evade such security countermeasures, attackers have used DNS agility, e.g., by using new domains daily to evade static blacklists and firewalls. In this paper we propose Notos, a dynamic reputation system for DNS. The premise of this system is that malicious, agile use of DNS has unique characteristics and can be distinguished from legitimate, professionally provisioned DNS services. Notos uses passive DNS query data and analyzes the network and zone features of domains. It builds models of known legitimate domains and malicious domains, and uses these models to compute a reputation score for a new domain indicative of whether the domain is malicious or legitimate. We have evaluated Notos in a large ISP's network with DNS traffic from 1.4 million users. Our results show that Notos can identify malicious domains with high accuracy (true positive rate of 96.8%) and low false positive rate (0.38%), and can identify these domains weeks or even months before they appear in public blacklists. <s> BIB002 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Clustering-Based Methods <s> As more and more Internet-based attacks arise, organizations are responding by deploying an assortment of security products that generate situational intelligence in the form of logs. These logs often contain high volumes of interesting and useful information about activities in the network, and are among the first data sources that information security specialists consult when they suspect that an attack has taken place. However, security products often come from a patchwork of vendors, and are inconsistently installed and administered. They generate logs whose formats differ widely and that are often incomplete, mutually contradictory, and very large in volume. Hence, although this collected information is useful, it is often dirty. We present a novel system, Beehive, that attacks the problem of automatically mining and extracting knowledge from the dirty log data produced by a wide variety of security products in a large enterprise. We improve on signature-based approaches to detecting security incidents and instead identify suspicious host behaviors that Beehive reports as potential security incidents. These incidents can then be further analyzed by incident response teams to determine whether a policy violation or attack has occurred. We have evaluated Beehive on the log data collected in a large enterprise, EMC, over a period of two weeks. We compare the incidents identified by Beehive against enterprise Security Operations Center reports, antivirus software alerts, and feedback from enterprise security specialists. We show that Beehive is able to identify malicious events and policy violations which would otherwise go undetected. <s> BIB003 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Clustering-Based Methods <s> Peer-to-peer (P2P) botnets have recently been adopted by botmasters for their resiliency against take-down efforts. Besides being harder to take down, modern botnets tend to be stealthier in the way they perform malicious activities, making current detection approaches ineffective. In addition, the rapidly growing volume of network traffic calls for high scalability of detection systems. In this paper, we propose a novel scalable botnet detection system capable of detecting stealthy P2P botnets. Our system first identifies all hosts that are likely engaged in P2P communications. It then derives statistical fingerprints to profile P2P traffic and further distinguish between P2P botnet traffic and legitimate P2P traffic. The parallelized computation with bounded complexity makes scalability a built-in feature of our system. Extensive evaluation has demonstrated both high detection accuracy and great scalability of the proposed system. <s> BIB004 </s> On the Security of Machine Learning in Malware C&C Detection: A Survey <s> Clustering-Based Methods <s> Abstract Botnets continue to be used by attackers to perform various malicious activities on the Internet. Over the past years, many botnet detection techniques have been proposed; however, most of them cannot detect botnets in an early stage of their lifecycle, or they often depend on a specific command and control protocol. In this paper, we propose BotGrab, a general botnet detection system that considers both malicious activities and the history of coordinated group activities in the network to identify bot-infected hosts. BotGrab tracks suspected hosts participating in some coordinated group activities and calculates a negative reputation score for each of them based on the history of their participation in these activities. A suspected host will be identified as being bot-infected if it has a high negative reputation score or performs some malicious activities while having a low negative reputation score. We demonstrate the effectiveness of BotGrab to detect various botnets including HTTP-, IRC-, and P2P-based botnets using a testbed network consisting of some bot-infected hosts. <s> BIB005
The main disadvantage of using a signature-or classifier-based detection method is that these systems are usually not as effective at detecting new, or updated, malware due to an inherent assumption of stationary data. Considering that malware changes frequently, every time a new variant of malware is discovered, or an existing piece updates itself, the signatures have to be recreated or the classifier retrained. If the new variant is not discovered, then it is unlikely to be detected by these systems. Clustering-based systems can account for unknown behavior. In these systems, the algorithms attempt to separate different patterns of behavior without necessarily knowing itself what is malicious or benign. The output of clustering is often used to produce signatures. One particular issue in detecting malware in large organizations is the problem of "dirty" logs. This refers to the large variety in logs that are kept by different systems that are often incompatible and inconsistent, and may also contain a large amount of duplication between different sets of logs. Security information and event management (SIEM) systems have been normalizing logs for several years, and this practice has been brought to the detection space. In Beehive BIB003 , log data is first normalized. For example, hosts are all mapped to IP addresses, and time stamps are all converted to UTC. A set of features are then extracted for each host, which can be tuned to the setting, but as an example contain traffic volume information and enterprise policy breaches (e.g., accessing blocked pages). An iterative variant of k-means clustering is then applied, which results in a set of outliers that are then labeled as suspicious. These hosts are then subject to manual investigation. When tested on a large enterprise network, the system produced alerts that were 25% malicious, 45% policy violations, and 35% labeled as other. BIB004 propose a system for detecting P2P botnets that differs from previous attempts in that it is unsupervised, meaning that no knowledge of existing malicious behavior is required. The system first applies clustering of flows to identify hosts that have taken part in P2P activity, and then it applies two layers of filtering and then hierarchical clustering to group hosts that are taking part in a malicious P2P network. They show that it can separate malicious P2P traffic from benign with a 100% TP rate and a low 0.02% FP rate. In BotGrab, BIB005 also cluster flows using a custom online flow clustering algorithm but then apply a reputation engine to identify hosts that have a negative reputation. Reputation is based on participation in coordinated activities, an indication of botnet membership. The reputation engine can be combined with knowledge of hosts participating in malicious activities (e.g., DDoS) to achieve a TP rate of at least 97% and an FP rate of at most 2.3% (if the knowledge of malicious activities is not used, this is reduced to 92% and 2.02%, respectively). BIB001 propose TĀMD, a system for identifying candidate groups of infected computers within a network by aggregating similar flows. Three aggregators extract aggregates of hosts based on communication destinations, packet payloads, and the host platform (OS). In particular, the destination aggregator makes use of a k-means clustering variant. This system is able to identify all malicious hosts in most cases (when VM outputs are inserted into background traffic), except for an IRC botnet, which achieved an 87% detection rate. 4.5.1. DNS Traffic Analysis. We have already discussed signature-and classifier-based detection systems that make use of DNS traffic analysis. We will now explore systems that use clustering-based methods. One proposed detection method is to make use of the reputation of domain names to decide if they are related to malicious activities BIB002 . In this system (Notos), domains are clustered first by the IP addresses associated with them and second according to similarities in the syntactic structure of the domain names themselves. The k-means clustering algorithm is used. These clusters are then classified as malicious or not based on a collection of whitelists and blacklists: domains in a cluster that contains blacklist domains are likely to be malicious themselves. This system is run on local DNS servers and can achieve a TP rate of 96% and a low FP rate. In an attempt to identify domains accessed as a result of a domain generation algorithm (DGA),  apply clustering to domains extracted from DNS queries. The system first filters and removes domains that are human pronounceable. The remaining domains are then clustered with those collected from blacklists using the DBSCAN clustering algorithm, using features related to the IPs to which the domains resolve. These clusters are used to extract fingerprints that are used to match new domains to known DGAs. When tested on a 3-month dataset taken from the DNSDB, with Conficker, Torpig, and Bamital domains inserted, the system could filter out 50%, 35%, and 62% of the DGA domains, respectively, resulting in recall between 81% and 94%. The system also contains an intelligence module that is able to track the evolution of IPs to which groups of domains point to monitor changes in botnet behavior.