Indicators of Compromise (IOCs) are artifacts observed on a network or in an operating system that can be utilized to indicate a computer intrusion and detect cyber-attacks in an early stage. Thus, they exert an important role in the field of cybersecurity. However, state-of-the-art IOCs detection systems rely heavily on hand-crafted features with expert knowledge of cybersecurity, and require large-scale manually annotated corpora to train an IOC classifier. In this paper, we propose using an end-to-end neural-based sequence labelling model to identify IOCs automatically from cybersecurity articles without expert knowledge of cybersecurity. By using a multi-head self-attention module and contextual features, we find that the proposed model is capable of gathering contextual information from texts of cybersecurity articles and performs better in the task of IOC identification. Experiments show that the proposed model outperforms other sequence labelling models, achieving the average F1-score of 89.0 on English cybersecurity article test set, and approximately the average F1-score of 81.8 on Chinese test set.
Few references in cyber security utilize natural language processing. Neuhaus and Zimmermann @cite_26 analyze the trend of vulnerability by applying latent Dirichlet allocation to vulnerability description. @cite_14 put forward a system to automatically extract IOC items from blog posts. @cite_4 proposed a system that automatically extracted threat actions from unstructured threat intelligence reports by utilizing a pre-defined ontology. A concurrent work by @cite_24 automatically extracted IOC data from security technical articles and further categorized them into different stages of malicious campaigns. All of those systems consist of several components that rely heavily on manually defined rules, while our proposed model is an end-to-end model using word embedding and token features as input, which is more general and applicable to a broader area.
@cite_24 @cite_14 @cite_26 @cite_4
2538865281 2837911466 2799725809 2171369953
To adapt to the rapidly evolving landscape of cyber threats, security professionals are actively exchanging Indicators of Compromise (IOC) (e.g., malware signatures, botnet IPs) through public sources (e.g. blogs, forums, tweets, etc.). Such information, often presented in articles, posts, white papers etc., can be converted into a machine-readable OpenIOC format for automatic analysis and quick deployment to various security mechanisms like an intrusion detection system. With hundreds of thousands of sources in the wild, the IOC data are produced at a high volume and velocity today, which becomes increasingly hard to manage by humans. Efforts to automatically gather such information from unstructured text, however, is impeded by the limitations of today's Natural Language Processing (NLP) techniques, which cannot meet the high standard (in terms of accuracy and coverage) expected from the IOCs that could serve as direct input to a defense system. In this paper, we present iACE, an innovation solution for fully automated IOC extraction. Our approach is based upon the observation that the IOCs in technical articles are often described in a predictable way: being connected to a set of context terms (e.g., "download") through stable grammatical relations. Leveraging this observation, iACE is designed to automatically locate a putative IOC token (e.g., a zip file) and its context (e.g., "malware", "download") within the sentences in a technical article, and further analyze their relations through a novel application of graph mining techniques. Once the grammatical connection between the tokens is found to be in line with the way that the IOC is commonly presented, these tokens are extracted to generate an OpenIOC item that describes not only the indicator (e.g., a malicious zip file) but also its context (e.g., download from an external source). Running on 71,000 articles collected from 45 leading technical blogs, this new approach demonstrates a remarkable performance: it generated 900K OpenIOC items with a precision of 95 and a coverage over 90 , which is way beyond what the state-of-the-art NLP technique and industry IOC tool can achieve, at a speed of thousands of articles per hour. Further, by correlating the IOCs mined from the articles published over a 13-year span, our study sheds new light on the links across hundreds of seemingly unrelated attack instances, particularly their shared infrastructure resources, as well as the impacts of such open-source threat intelligence on security protection and evolution of attack strategies. Modern cyber attacks consist of a series of steps and are generally part of larger campaigns. Large-scale field data provides a quantitative measurement of these campaigns. On the other hand, security practitioners extract and report qualitative campaign characteristics manually. Linking the two sources provides new insights about attacker strategies from measurements. However, this is a time-consuming task because qualitative measurements are generally reported in natural language and are not machine-readable. We propose an approach to bridge measurement data with manual analysis. We borrow the idea from threat intelligence: we define campaigns using a 4-stage model, and describe each stage using IOCs (indicators of compromise), e.g. URLs and IP addresses. We train a multi-class classifier to extract IOCs and further categorize them into different stages. We implement these ideas in a system called ChainSmith. Our system can achieve 91.9 precision and 97.8 recall in extracting IOCs, and can determine the campaign roles for 86.2 of IOCs with 78.2 precision and 80.7 recall. We run ChainSmith on 14,155 online security articles, from which we collect 24,653 IOCs. The semantic roles allow us to link manual attack analysis with large scale field measurements. In particular, we study the effectiveness of different persuasion techniques used on enticing user to download the payloads. We find that the campaign usually starts from social engineering and "missing codec" ruse is a common persuasion technique that generates the most suspicious downloads each day. Abstract Security information and event management (SIEM) systems receive a large number of alerts from different intrusion detection systems. They are expected, from these alerts, to make reliable and timely decisions regarding the types of ongoing attack scenarios and their priorities. However, the lack of an agreed-upon vocabulary for the representation of the domain knowledge makes it difficult for state-of-the-art SIEM systems to effectively manage these complex decisions. To overcome this problem, an ontology-based expert system approach can provide domain knowledge modeling as a foundation for disambiguation of meaning and automatic reasoning regarding ongoing attack scenarios. The proposed approach reconstructs attack scenarios by reasoning based on the evidences in the alert stream. The main idea of the proposed approach is to identify the causal relation between alerts using their similarity. This approach assumes that the similarity between two successive steps in an attack scenario is greater than that of two non-successive steps. Moreover, the similarity between the steps of the same attack scenario is greater than that between the steps of two different attack scenarios. The benefit of the proposed approach includes the fast and incremental reconstruction of known and unknown attack scenarios without expert intervention, which is an enormous step forward in developing expert and intelligent systems for cyber security. We evaluated the proposed technique by performing experiments on two known datasets: DARPA 2000 and MACCDC 2012. The results prove the advantages of the proposed approach with regard to completeness and soundness criteria. The ability of terrorist networks to conduct sophisticated and simultaneous attacks - the most recent one on March 11, 2004 in Madrid, Spain - suggests that there is a significant need for developing information technology tools for counter-terrorism analysis. These technologies could empower intelligence analysts to find information faster, share, and collaborate across agencies, "connect the dots" better, and conduct quicker and better analyses. One such technology, the adaptive safety analysis and monitoring (ASAM) system, is under development at the University of Connecticut. In this paper, the ASAM system is introduced and its capabilities are discussed. The vulnerabilities at the Athens 2004 Olympics are modeled and patterns of anomalous behavior are identified using a combination of feature-aided multiple target tracking, hidden Markov models (HMMs), and Bayesian networks (BNs). Functionality of the ASAM system is illustrated by way of application to two hypothetical models of terrorist activities at the Athens 2004 Olympics.