TITLE: A Survey on Industrial Control System Testbeds and Datasets for Security Research 

ABSTRACT: The increasing digitization and interconnection of legacy Industrial Control
Systems (ICSs) open new vulnerability surfaces, exposing such systems to
malicious attackers. Furthermore, since ICSs are often employed in critical
infrastructures (e.g., nuclear plants) and manufacturing companies (e.g.,
chemical industries), attacks can lead to devastating physical damages. In
dealing with this security requirement, the research community focuses on
developing new security mechanisms such as Intrusion Detection Systems (IDSs),
facilitated by leveraging modern machine learning techniques. However, these
algorithms require a testing platform and a considerable amount of data to be
trained and tested accurately. To satisfy this prerequisite, Academia,
Industry, and Government are increasingly proposing testbed (i.e., scaled-down
versions of ICSs or simulations) to test the performances of the IDSs.
Furthermore, to enable researchers to cross-validate security systems (e.g.,
security-by-design concepts or anomaly detectors), several datasets have been
collected from testbeds and shared with the community. In this paper, we
provide a deep and comprehensive overview of ICSs, presenting the architecture
design, the employed devices, and the security protocols implemented. We then
collect, compare, and describe testbeds and datasets in the literature,
highlighting key challenges and design guidelines to keep in mind in the design
phases. Furthermore, we enrich our work by reporting the best performing IDS
algorithms tested on every dataset to create a baseline in state of the art for
this field. Finally, driven by knowledge accumulated during this survey's
development, we report advice and good practices on the development, the
choice, and the utilization of testbeds, datasets, and IDSs.
The increasing digitization and interconnection of legacy Industrial Control
Systems (ICSs) open new vulnerability surfaces, exposing such systems to
malicious attackers. Furthermore, since ICSs are often employed in critical
infrastructures (e.g., nuclear plants) and manufacturing companies (e.g.,
chemical industries), attacks can lead to devastating physical damages. In
dealing with this security requirement, the research community focuses on
developing new security mechanisms such as Intrusion Detection Systems (IDSs),
facilitated by leveraging modern machine learning techniques. However, these
algorithms require a testing platform and a considerable amount of data to be
trained and tested accurately. To satisfy this prerequisite, Academia,
Industry, and Government are increasingly proposing testbed (i.e., scaled-down
versions of ICSs or simulations) to test the performances of the IDSs.
Furthermore, to enable researchers to cross-validate security systems (e.g.,
security-by-design concepts or anomaly detectors), several datasets have been
collected from testbeds and shared with the community. In this paper, we
provide a deep and comprehensive overview of ICSs, presenting the architecture
design, the employed devices, and the security protocols implemented. We then
collect, compare, and describe testbeds and datasets in the literature,
highlighting key challenges and design guidelines to keep in mind in the design
phases. Furthermore, we enrich our work by reporting the best performing IDS
algorithms tested on every dataset to create a baseline in state of the art for
this field. Finally, driven by knowledge accumulated during this survey's
development, we report advice and good practices on the development, the
choice, and the utilization of testbeds, datasets, and IDSs.
A Survey on Industrial Control System Testbeds and Datasets for Security Research
http://arxiv.org/abs/2102.05631v3