A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> This paper presents the preliminary architecture of a network level intrusion detection system. The proposed system will monitor base level information in network packets (source, destination, packet size, and time), learning the normal patterns and announcing anomalies as they occur. The goal of this research is to determine the applicability of current intrusion detection technology to the detection of network level intrusions. In particular, the authors are investigating the possibility of using this technology to detect and react to worm programs. <s> BIB001 </s> A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> In this paper we discuss our research in developing general and systematic methods for intrusion detection. The key ideas are to use data mining techniques to discover consistent and useful patterns of system features that describe program and user behavior, and use the set of relevant system features to compute (inductively learned) classifiers that can recognize anomalies and known intrusions. Using experiments on the sendmail system call data and the network tcpdump data, we demonstrate that we can construct concise and accurate classifiers to detect anomalies. We provide an overview on two general data mining algorithms that we have implemented: the association rules algorithm and the frequent episodes algorithm. These algorithms can be used to compute the intra-and inter-audit record patterns, which are essential in describing program or user behavior. The discovered patterns can guide the audit data gathering process and facilitate feature selection. To meet the challenges of both efficient learning (mining) and real-time detection, we propose an agent-based architecture for intrusion detection systems where the learning agents continuously compute and provide the updated (detection) models to the detection agents. <s> BIB002 </s> A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> Network intrusion detection systems (NIDS) are an important part of any network security architecture. They provide a layer of defense which monitors network traffic for predefined suspicious activity or patterns, and alert system administrators when potential hostile traffic is detected. Commercial NIDS have many differences, but Information Systems departments must face the commonalities that they share such as significant system footprint, complex deployment and high monetary cost. Snort was designed to address these issues. <s> BIB003 </s> A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> Intrusion detection systems have traditionally been based on the characterization of an attack and the tracking of the activity on the system to see if it matches that characterization. Recently, new intrusion detection systems based on data mining are making their appearance in the field. This paper describes the design and experiences with the ADAM (Audit Data Analysis and Mining) system, which we use as a testbed to study how useful data mining techniques can be in intrusion detection. <s> BIB004 </s> A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> Intrusion detection corresponds to a suite of techniques that are used to identify attacks against computers and network infrastructures. Anomaly detection is a key element of intrusion detection in which perturbations of normal behavior suggest the presence of intentionally or unintentionally induced attacks, faults, defects, etc. This paper focuses on a detailed comparative study of several anomaly detection schemes for identifying different network intrusions. Several existing supervised and unsupervised anomaly detection schemes and their variations are evaluated on the DARPA 1998 data set of network connections [9] as well as on real network data using existing standard evaluation techniques as well as using several specific metrics that are appropriate when detecting attacks that involve a large number of connections. Our experimental results indicate that some anomaly detection schemes appear very promising when detecting novel intrusions in both DARPA’98 data and real network data. <s> BIB005 </s> A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> Anomaly detection is an important problem that has been researched within diverse research areas and application domains. Many anomaly detection techniques have been specifically developed for certain application domains, while others are more generic. This survey tries to provide a structured and comprehensive overview of the research on anomaly detection. We have grouped existing techniques into different categories based on the underlying approach adopted by each technique. For each category we have identified key assumptions, which are used by the techniques to differentiate between normal and anomalous behavior. When applying a given technique to a particular domain, these assumptions can be used as guidelines to assess the effectiveness of the technique in that domain. For each category, we provide a basic anomaly detection technique, and then show how the different existing techniques in that category are variants of the basic technique. This template provides an easier and more succinct understanding of the techniques belonging to each category. Further, for each category, we identify the advantages and disadvantages of the techniques in that category. We also provide a discussion on the computational complexity of the techniques since it is an important issue in real application domains. We hope that this survey will provide a better understanding of the different directions in which research has been done on this topic, and how techniques developed in one area can be applied in domains for which they were not intended to begin with. <s> BIB006 </s> A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> Distributed denial of service attacks are the serious candidates for traffic analysis next to traffic performance evaluation. As these threats deplete the network resources rapidly particularly link parameters, modeling these attacks provide a strong base for analyzing the attack characteristics. The solution domain uses active networks for implementation, as it supports active routers which can perform customized tasks on demand and ease of deploying. The paper presents a model based on packet attributes to characterize the attack traffic and a detection and response framework based on the model. The detection mechanism uses leaky buckets to rate limit the traffic based on the packet ranking using linear arithmetic. The simulation results depicting the attack traffic passed through the network as well as the legitimate traffic dropped at the active routers, under different attack scenarios, are found to be comparable to existing solutions with improved efficiency in detection rate and time. <s> BIB007 </s> A Survey of Outlier Detection Methods in Network Anomaly Identification <s> Outlier Detection in Anomaly Detection <s> The main function of IDS (Intrusion Detection System) is to protect the system, analyze and predict the behaviors of users. Then these behaviors will be considered an attack or a normal behavior. Though IDS has been developed for many years, the large number of return alert messages makes managers maintain system inefficiently. In this paper, we use RST (Rough Set Theory) and SVM (Support Vector Machine) to detect intrusions. First, RST is used to preprocess the data and reduce the dimensions. Next, the features selected by RST will be sent to SVM model to learn and test respectively. The method is effective to decrease the space density of data. The experiments will compare the results with different methods and show RST and SVM schema could improve the false positive rate and accuracy. <s> BIB008
The anomaly detection problem is similar to the problem of finding outliers, specifically, in network intrusion detection. Intrusion detection is a part of a security management system for computers and networks. Intrusion BIB001 is a set of actions aimed to compromise computer security goals such as confidentiality, integrity and availability of resources. Traditional technologies such as firewalls are used to build a manual passive defence system against attacks. An Intrusion Detection System (IDS) is usually used to enhance the network security of enterprises by monitoring and analysing network data packets. Intrusion detection is a system's "second line of defence" BIB008 . IDSs play a vital role in network security. Network intrusion detection systems (NIDSs) can detect attacks by observing network activities. Intrusion detection techniques are used, primarily, for misuse detection and anomaly detection. Misuse based detection involves an attempt to define a set of rules (also called signatures) that can be used to decide that a given behavior is that of an intruder. For example, Snort BIB003 is a misuse based NIDS. The other approach, anomaly detection, involves the collection of data relating to the behavior of legitimate users over a period of time, and then applying tests to the gathered data to determine whether that behavior is legitimate user behavior or not. Anomaly detection has the advantage that it can detect new attacks that the system has never seen before as they deviate from normal behavior. ADAM BIB004 is a well known anomaly detection NIDS. The key challenge for outlier detection in this domain is the huge volume of data. Outlier detection schemes need to be computationally efficient to handle these large sized inputs. An outlier can be an observation that is distinctly different or is at a position of abnormal distance from other values in the dataset. Detection of abnormal behavior can be based on features extracted from traces such as network trace or system call trace BIB002 . An intrusion can be detected by finding an outlier whose features are distinctly different from the rest of the data. Outliers can often be individuals or groups of clients exhibiting behavior outside the range of what is considered normal. In order to apply outlier detection to anomaly based network intrusion detection, it is assumed BIB006 that -1. The majority of the network connections are normal traffic. Only a small amount of traffic is malicious. 2. Attack traffic is statistically different from normal traffic. However, in a real-world network scenario, these assumptions may not be always true. For example, when dealing with DDoS (distributed denial of service) BIB007 or bursty attack BIB005 detection in computer networks, the anomalous traffic is actually more frequent than the normal traffic.