A Survey of Physics-Based Attack Detection in Cyber-Physical Systems <s> Ignoring False Alarms Does not Provide a Complete Evaluation. <s> Classification accuracy in intrusion detection systems (IDSs) deals with such fundamental problems as how to compare two or more IDSs, how to evaluate the performance of an IDS, and how to determine the best configuration of the IDS. In an effort to analyze and solve these related problems, evaluation metrics such as the Bayesian detection rate, the expected cost, the sensitivity and the intrusion detection capability have been introduced. In this paper, we study the advantages and disadvantages of each of these performance metrics and analyze them in a unified framework. Additionally, we introduce the intrusion detection operating characteristic (IDOC) curves as a new IDS performance tradeoff which combines in an intuitive way the variables that are more relevant to the intrusion detection evaluation problem. We also introduce a formal framework for reasoning about the performance of an IDS and the proposed metrics against adaptive adversaries. We provide simulations and experimental results to illustrate the benefits of the proposed framework. <s> BIB001 </s> A Survey of Physics-Based Attack Detection in Cyber-Physical Systems <s> Ignoring False Alarms Does not Provide a Complete Evaluation. <s> A power grid is a complex system connecting electric power generators to consumers through power transmission and distribution networks across a large geographical area. System monitoring is necessary to ensure the reliable operation of power grids, and state estimation is used in system monitoring to best estimate the power grid state through analysis of meter measurements and power system models. Various techniques have been developed to detect and identify bad measurements, including interacting bad measurements introduced by arbitrary, nonrandom causes. At first glance, it seems that these techniques can also defeat malicious measurements injected by attackers. In this article, we expose an unknown vulnerability of existing bad measurement detection algorithms by presenting and analyzing a new class of attacks, called false data injection attacks, against state estimation in electric power grids. Under the assumption that the attacker can access the current power system configuration information and manipulate the measurements of meters at physically protected locations such as substations, such attacks can introduce arbitrary errors into certain state variables without being detected by existing algorithms. Moreover, we look at two scenarios, where the attacker is either constrained to specific meters or limited in the resources required to compromise meters. We show that the attacker can systematically and efficiently construct attack vectors in both scenarios to change the results of state estimation in arbitrary ways. We also extend these attacks to generalized false data injection attacks, which can further increase the impact by exploiting measurement errors typically tolerated in state estimation. We demonstrate the success of these attacks through simulation using IEEE test systems, and also discuss the practicality of these attacks and the real-world constraints that limit their effectiveness. <s> BIB002 </s> A Survey of Physics-Based Attack Detection in Cyber-Physical Systems <s> Ignoring False Alarms Does not Provide a Complete Evaluation. <s> Cyber-physical systems integrate computation, communication, and physical capabilities to interact with the physical world and humans. Besides failures of components, cyber-physical systems are prone to malignant attacks, and specific analysis tools as well as monitoring mechanisms need to be developed to enforce system security and reliability. This paper proposes a unified framework to analyze the resilience of cyber-physical systems against attacks cast by an omniscient adversary. We model cyber-physical systems as linear descriptor systems, and attacks as exogenous unknown inputs. Despite its simplicity, our model captures various real-world cyber-physical systems, and it includes and generalizes many prototypical attacks, including stealth, (dynamic) false-data injection and replay attacks. First, we characterize fundamental limitations of static, dynamic, and active monitors for attack detection and identification. Second, we provide constructive algebraic conditions to cast undetectable and unidentifiable attacks. Third, by using the system interconnection structure, we describe graph-theoretic conditions for the existence of undetectable and unidentifiable attacks. Finally, we validate our findings through some illustrative examples with different cyber-physical systems, such as a municipal water supply network and two electrical power grids. <s> BIB003 </s> A Survey of Physics-Based Attack Detection in Cyber-Physical Systems <s> Ignoring False Alarms Does not Provide a Complete Evaluation. <s> While attacks on information systems have for most practical purposes binary outcomes (information was manipulated/eavesdropped, or not), attacks manipulating the sensor or control signals of Industrial Control Systems (ICS) can be tuned by the attacker to cause a continuous spectrum in damages. Attackers that want to remain undetected can attempt to hide their manipulation of the system by following closely the expected behavior of the system, while injecting just enough false information at each time step to achieve their goals. In this work, we study if attack-detection can limit the impact of such stealthy attacks. We start with a comprehensive review of related work on attack detection schemes in the security and control systems community. We then show that many of those works use detection schemes that are not limiting the impact of stealthy attacks. We propose a new metric to measure the impact of stealthy attacks and how they relate to our selection on an upper bound on false alarms. We finally show that the impact of such attacks can be mitigated in several cases by the proper combination and configuration of detection schemes. We demonstrate the effectiveness of our algorithms through simulations and experiments using real ICS testbeds and real ICS systems. <s> BIB004
As we discussed before, the line of research started by false data injection attacks for state estimation in the power grid BIB002 BIB002 focuses on developing new ways to find attacks or to find new undetectable attacks; however, they tend to ignore the evaluation of the system under normal conditions (the false-alarm rate). A similar emphasis on attack detection and on identifying undetectable attacks but ignoring false alarms can be seen in the control theory community BIB003 ; at the end of the day, you can detect all attacks by generating an alert at every single time-step k, but this will give raise to an unmanageable number of false alarms. Lessons From The Last Three Attacks in Section 6.1. If we had evaluated our anomaly detection algorithm against using a traditional intrusion detection metric like ROC curves, and our attack examples consisted of the last three attacks presented in the previous section (a stealthy attacker), then we would have had a 0% detection rate; that is, our ROC curve would be a flat line along the x-axis with a 0% value in the y-axis (Figure 17 (left) ). This problem is not unique to ROC curves, most popular metrics for evaluating the classification accuracy of intrusion detection systems can be shown to be a multi-criteria optimization problem between the false-alarm rate, and the true positive rate BIB001 , and all of them depend on the ability of a system to detect some attacks. Fig. 15 . Difference between a fault and an attack: a sophisticated attacker will remain undetected by maintaining the anomaly detection statistic S below the threshold τ to avoid raising alarms. To obtain the true positive rate of a detection algorithm, we need to generate an attack that will be detected, and it is not clear if there is a principled way of justifying that to evaluate a system we need to generate attacks that will be detected, as this implies that the adversary is not adaptive and will not attempt to evade our detection algorithms. In the previous section, we showed that for any anomaly threshold τ , a "smart" attacker can always launch an attack that keeps the anomaly detection statistic below this threshold, and therefore this "smart" attacker can always launch attacks that will not be detected (i.e., the attacker can create a variety of attacks that will have a 0% detection rate). Figure 15 illustrates this problem. In this figure, an anomaly detection statistic S keeps score of the "anomalous" state in the system: if S increases beyond the threshold τ , it will raise an alarm. Random failures are expected to increase the anomaly score, but a sophisticated attacker that knows about this anomaly detection test will be able to remain undetected. The question that we need to answer here is then, How much can the attacker affect the system while remaining undetected? In addition to a metric that quantifies how much the attacker can affect the system without being detected, we need to consider a metric that shows the trade-offs involved. Most of the work in control theory and power system conferences ignore false-alarm rates in their analyses BIB002 BIB002 BIB003 ; however, at the end of the day, you can detect all attacks by generating an alert at every single time-step k, but this will give rise to an unmanageable number of false alarms, so we need to illustrate the inherent trade-off between security and false alarms (usability). In conclusion, the traditional trade-off between false alarms and detection rate is not a good fit for our problem; however, focusing solely on model fidelity will not give us any indication of what an attacker can do. Ignoring false alarms prevents assessment of the practicality and usability of the system. Design Options for Metrics. Looking again at our literature review, the majority of previous work uses a model of the physical system (LDS or AR) to generate an expected valueŷ k . This prediction is then compared to the sensor measurements y k to generate a residual r k = |ŷ k − y k |. We test if r k > τ , where τ is a threshold we can adjust to lower false alarms while still hoping to achieve good detection. A stateless test generates an alarm if r k > τ , where τ is a threshold we can adjust to lower false alarms while still hoping to achieve good detection. A stateful test instead will compute an additional statistic S k that keeps track of the historical changes of r k and will generate an alert if S k ≥ τ (another appropriately chosen threshold). We can clearly see that increasing the threshold will reduce the number of false alarms; however, what do we give up by reducing the number of false alarms? Traditionally the trade-off for reducing Fig. 16 . Illustration of our proposed tradeoff metric in Reference BIB004 . The y-axis is a measure of the worst the attacker can do while remaining undetected, and the x-axis represents the expected time between false alarms E[T f a ]. Anomaly detection algorithms are then evaluated for different points in this space. the number of false alarms is a reduced true positive rate, but as we discussed before, this is not a good metric for our case. Notice that, if the threshold is too low, an attacker has to produce attacks where y k will be similar from the expected behavior of our models, but if it is too high, the attacker has more leeway to deviate y k and damage to the system without raising alarms. We argue that the metric that we need is one that shows the trade-off between the number of false alarms and the ability to minimize the negative consequences of undetected attacks.