Cyber-security is an important societ al concern. Cyber-attacks have increased in numbers as well as in the extent of damage caused in every attack. Large organizations operate a Cyber Security Operation Center (CSOC), which form the first line of cyber-defense. The inspection of cyber-alerts is a critical part of CSOC operations. A recent work, in collaboration with Army Research Lab, USA proposed a reinforcement learning (RL) based approach to prevent the cyber-alert queue length from growing large and overwhelming the defender. Given the potential deployment of this approach to CSOCs run by US defense agencies, we perform a red team (adversarial) evaluation of this approach. Further, with the recent attacks on learning systems, it is even more important to test the limits of this RL approach. Towards that end, we learn an adversarial alert generation policy that is a best response to the defender inspection policy. Surprisingly, we find the defender policy to be quite robust to the best response of the attacker. In order to explain this observation, we extend the earlier RL model to a game model and show that there exists defender policies that can be robust against any adversarial policy. We also derive a competitive baseline from the game theory model and compare it to the RL approach. However, we go further to exploit assumptions made in the MDP in the RL model and discover an attacker policy that overwhelms the defender. We use a double oracle approach to retrain the defender with episodes from this discovered attacker policy. This made the defender robust to the discovered attacker policy and no further harmful attacker policies were discovered. Overall, the adversarial RL and double oracle approach in RL are general techniques that are applicable to other RL usage in adversarial environments.
Game theoretic inspection or auditing has appeared in many papers @cite_14 @cite_8 @cite_27 , however, all these works have a single shot interaction model and are not focused on cyber-alert inspection. As stated already, one game based work @cite_9 on cyber-alert inspection is single shot, and has other stringent assumptions such as fixed number of analysts and alerts per hour. Another recent game theoretic work @cite_3 uses a zero-sum Markov game model for cyber-alert inspection, but the model assumes complete information for both the players and hence solves the game using standard minimax value iteration. Our approach accounts for the realistic scenario of incomplete information and signals, and explains theoretically and empirically the subtle relation between the budget of the players. Game theory has also been used for other problems in cyber-security such as deception @cite_2 , attack graphs @cite_11 , man in the middle attacks @cite_4 , and spear fishing @cite_6 @cite_17 , which are quite different from our cyber-security problem. Also, scalability is still an issue in solving partially observable stochastic games @cite_29 . Further, our game has a non-standard long term utility.
@cite_14 @cite_4 @cite_8 @cite_9 @cite_29 @cite_17 @cite_3 @cite_6 @cite_27 @cite_2 @cite_11
69361866 2141442361 2548493877 266047537
Abstract Starting with the analysis of arms control and disarmament problems in the sixties, inspection games have evolved into a special area of game theory with specific theoretical aspects, and, equally important, practical applications in various fields of human activity where inspection is mandatory. In this contribution, a survey of applications is given first. These include arms control and disarmament, theoretical approaches to auditing and accounting, for example in insurance, and problems of environmental surveillance. Then, the general problem of inspection is presented in a game-theoretic framework that extends a statistical hypothesis testing problem. This defines a game since the data can be strategically manipulated by an inspectee who wants to conceal illegal actions. Using this framework, two models are solved, which are practically significant and technically interesting: material accountancy and data verification. A second important aspect of inspection games is the fact that inspection resources are limited and have to be used strategically. This is demonstrated in the context of sequential inspection games, where many mathematically challenging models have been studied. Finally, the important concept of leadership, where the inspector becomes a leader by announcing and committing himself to his strategy, is shown to apply naturally to inspection games. This paper presents a novel decision-theoretic approach to control and coordinate multiple active cameras for observing a number of moving targets in a surveillance system. This approach offers the advantages of being able to (a) account for the stochasticity of targets' motion via probabilistic modeling, and (b) address the trade-off between maximizing the expected number of observed targets and the resolution of the observed targets through stochastic optimization. One of the key issues faced by existing approaches in multi-camera surveillance is that of scalability with increasing number of targets. We show how its scalability can be improved by exploiting the problem structure: as proven analytically, our decision-theoretic approach incurs time that is linear in the number of targets to be observed during surveillance. As demonstrated empirically through simulations, our proposed approach can achieve high-quality surveillance of up to 50 targets in real time and its surveillance performance degrades gracefully with increasing number of targets. We also demonstrate our proposed approach with real AXIS 214 PTZ cameras in maximizing the number of Lego robots observed at high resolution over a surveyed rectangular area. The results are promising and clearly show the feasibility of our decision-theoretic approach in controlling and coordinating the active cameras in real surveillance system. We consider in this chapter a class of two-player nonzero-sum stochastic games with incomplete information, which is inspired by recent applications of game theory in network security. We develop fully distributed reinforcement learning algorithms, which require for each player a minimal amount of information regarding the other player. At each time, each player can be in an active mode or in a sleep mode. If †This material is based upon work supported in part by the U.S. Air Force Office of Scientific Research (AFOSR) under grant number AFOSR MURI FA9550-09-1-0249. D R A F T October 2, 2011, 1:59am D R A F T 2 HYBRID LEARNING IN STOCHASTIC GAMES AND ITS APPLICATION IN NETWORK SECURITY a player is in an active mode, she updates her strategy and estimates of unknown quantities using a specific pure or hybrid learning pattern. The players’ intelligence and rationality are captured by the weighted linear combination of different learning patterns. We use stochastic approximation techniques to show that, under appropriate conditions, the pure or hybrid learning schemes with random updates can be studied using their deterministic ordinary differential equation (ODE) counterparts. Convergence to state-independent equilibria is analyzed for special classes of games, namely, games with two actions, and potential games. Results are applied to network security games between an intruder and an administrator, where the noncooperative behaviors are well characterized by the features of distributed hybrid learning. Abstract : Many disarmament or arms-control agreements may be monitored by sample inspections. Unlike the usual sampling procedures, sampling for arms-control agreements must take into account the possibility that the statistical universe from which samples are to be drawn may be tampered with so as to decrease the probability of detection of a violation. A game-theoretic model is formulated for studying a sampling problem in which the inspector is allowed to examine a fixed number (usually small) of items or natural events (e.g., items from an assembly line under an agreement limiting military production, or seismic events under a nuclear test-ban agreement). It is assumed that the inspections are to be performed within a fixed time period or on a series of events of fixed length. Optimal sampling procedures are derived as functions of the number of inspections and the size of the statistical universe. Some variations on the model are briefly considered.