Abstract Flow-based data sets are necessary for evaluating network-based intrusion detection systems (NIDS). In this work, we propose a novel methodology for generating realistic flow-based network traffic. Our approach is based on Generative Adversarial Networks (GANs) which achieve good results for image generation. A major challenge lies in the fact that GANs can only process continuous attributes. However, flow-based data inevitably contain categorical attributes such as IP addresses or port numbers. Therefore, we propose three different preprocessing approaches for flow-based data in order to transform them into continuous values. Further, we present a new method for evaluating the generated flow-based network traffic which uses domain knowledge to define quality tests. We use the three approaches for generating flow-based network traffic based on the CIDDS-001 data set. Experiments indicate that two of the three approaches are able to generate high quality data.
In network intrusion detection, anomaly-based approaches in particular suffer from accurate evaluation, comparison, and deployment which originates from the scarcity of adequate datasets. Many such datasets are internal and cannot be shared due to privacy issues, others are heavily anonymized and do not reflect current trends, or they lack certain statistical characteristics. These deficiencies are primarily the reasons why a perfect dataset is yet to exist. Thus, researchers must resort to datasets that are often suboptimal. As network behaviors and patterns change and intrusions evolve, it has very much become necessary to move away from static and one-time datasets toward more dynamically generated datasets which not only reflect the traffic compositions and intrusions of that time, but are also modifiable, extensible, and reproducible. In this paper, a systematic approach to generate the required datasets is introduced to address this need. The underlying notion is based on the concept of profiles which contain detailed descriptions of intrusions and abstract distribution models for applications, protocols, or lower level network entities. Real traces are analyzed to create profiles for agents that generate real traffic for HTTP, SMTP, SSH, IMAP, POP3, and FTP. In this regard, a set of guidelines is established to outline valid datasets, which set the basis for generating profiles. These guidelines are vital for the effectiveness of the dataset in terms of realism, evaluation capabilities, total capture, completeness, and malicious activity. The profiles are then employed in an experiment to generate the desirable dataset in a testbed environment. Various multi-stage attacks scenarios were subsequently carried out to supply the anomalous portion of the dataset. The intent for this dataset is to assist various researchers in acquiring datasets of this kind for testing, evaluation, and comparison purposes, through sharing the generated datasets and profiles. Network traffic generators are widely used in networking research and they are validated by a very broad range of metrics (mainly traffic characteristics). In this paper we overview the state of the art of these metrics and unveil that there is no consensus in the research community how to validate these traffic generators and which metric to choose for validation purpose. This situation makes it extremely difficult to evaluate validation results and compare different traffic generators. We advocate the research for finding a common set of metrics for the validation and comparative evaluation of traffic generators. Future large-scale complex computing environments present challenges to the real-time intrusion detection systems (IDSs). In this paper, we design a prototype with hybrid software-enabled detection engine on the basis of our improved block-based neural network (BBNN), and integrate it with a high-frequency FPGA board to form a real-time intrusion detection system. The established prototype can seamlessly feed the large-scale NetFlow data obtained from Cisco routers directly into the improved BBNN based IDS. The corresponding BBNN structure and parameter settings have been improved and experimentally tested. Experimental performance comparisons have been conducted against four major schemes of Support Vector Machine (SVM) and Naive Bayes algorithm. The results show that the improved BBNN outperforms other algorithms with respect to the classification and detection performances. The false alarm rate is successfully reduced as low as 5.14 while the genuine detection rate 99.92 is still maintained.