Recently, there has been a surge in interest in safe and robust techniques within reinforcement learning (RL). 
 Current notions of risk in RL fail to capture the potential for systemic failures such as abrupt stoppages from system failures or surpassing of safety thresholds and the appropriate responsive controls in such instances. We propose a novel approach to fault-tolerance within RL in which the controller learns a policy can cope with adversarial attacks and random stoppages that lead to failures of the system subcomponents. The results of the paper also cover fault-tolerant (FT) control so that the controller learns to avoid states that carry risk of system failures. By demonstrating that the class of problems is represented by a variant of SGs, we prove the existence of a solution which is a unique fixed point equilibrium of the game and characterise the optimal controller behaviour. We then introduce a value function approximation algorithm that converges to the solution through simulation in unknown environments. Reinforcement learning (RL) provides the promise of adaptive agents being able to discover solutions merely through repeated interaction with their environment. RL has been deployed in a number of real-world settings in which, using RL, an adaptive agent learns to perform complex tasks, often in environments shared by human beings. Large scale factory industrial applications, traffic light control (Arel et al., 2010) , robotics (Deisenroth et al., 2013) and autonomous vehicles (Shalev-Shwartz et al., 2016) are notable examples of settings to which RL methods have been applied. Numerous automated systems are however, susceptible to failures and unanticipated outcomes. Moreover, many real-world systems amenable to RL suffer the potential for random stoppages and abrupt failures; actuator faults, failing mechanical system components, sensor failures are few such examples. In these settings, executing preprogrammed behaviours or policies that have been trained in idealised simulated environments can prove vastly inadequate for the task of ensuring the safe execution of tasks. Consequently, in the presence of such occurrences, the deployment of RL agents introduces a risk of catastrophic outcomes whenever the agent is required to act so as to avoid adverse outcomes in unseen conditions. The important question of how to control the system in a way that is both robust against systemic faults and, minimises the risk of faults or damage therefore arises. In response to the need to produce RL algorithms that execute tasks with safety guarantees, a significant amount of focus has recently been placed on safe execution, robust control and riskminimisation (Garcıa and Fernández, 2015) . Examples include H ∞ control (Morimoto and Doya, 2001) , coherent risk, conditional value at risk (Tamar et al., 2015) . In general, these methods introduce an objective 1 defined with an expectation measure that either penalises actions that lead to greater uncertainty or embeds a more pessimistic view of the world (for example, by biasing the transition predictions towards less desirable states). In both cases, the resulting policies act more cautiously over the horizon of the problem as compared to policies trained with a standard objective function. Despite the recent focus on safe methods within RL, the question of how to train an RL agent that can cope with random failures remains unaddressed. In particular, at present the question of how to produce an RL policy that can cope with an abrupt failure of some system subcomponent has received no systematic treatment. Similarly, the task of addressing how to produce RL policies that account for the risk of states in which such failures occur has not been addressed. In this paper, we for the first time produce a method that learns optimal policies in response to random and adversarial systems attacks that lead to stoppages of system (sub)components that may produce adverse events. Our method works by introducing an adversary that seeks to determine a stopping criterion to stop the system at states that lead to the worst possible (overall) outcomes for the controller. Using a game-theoretic construction, we then show how a policy that is robust against adversarial attacks that lead to abrupt failure can be learned by an adaptive agent using an RL updating method. In particular, the introduction of an adversary that performs attacks at states that lead to worst outcomes generates experiences for the adaptive RL agent to learn a best-response policy against such scenarios. To tackle this problem, we construct a novel two-player stochastic game (SG) in which one of the players, the controller, is delegated the task of learning to modify the system dynamics through its actions that maximise its payoff and an adversary or 'stopper' that enacts a strategy that stops the system in such a way that maximises the controller's costs. This produces a framework that finds optimal policies that are robust against stoppages at times that pose the greatest risk of catastrophe. The main contribution of the paper is to perform the first systematic treatment of the problem of robust control under worst-case failures. In particular, we perform a formal analysis of the game between the controller and the stopper. Our main results are centered around a minimax proof that establishes the existence of a value of the game. This is necessary for simulating the stopping action to induce fault-tolerance. Although minimax proofs are well-known in game theory (Shapley, 1953; Maitra and Parthasarathy, 1970; Filar et al., 1991) , replacing a player's action set with stopping rules necessitates a minimax proof (which now relies on a construction of open sets) which markedly differs to the standard methods within game theory. Additionally, crucial to our analysis is the characterisation of the adversary optimal stopping rule (Theorem 3). Our results tackle optimal stopping problems (OSPs) under worst-case transitions. OSPs are a subclass of optimal stochastic control (OSC) problems in which the goal is to determine a criterion for stopping at a time that maximises some state-dependent payoff (Peskir and Shiryaev, 2006) . The framework is developed through a series of theoretical results: first, we establish the existence of a value of the game which characterises the payoff for the saddle point equilibrium (SPE). Second, we prove a contraction mapping property of a Bellman operator of the game and that the value is a unique fixed point of the operator. Third, we prove the existence and characterise the optimal stopping time. We then prove an equivalence between the game of control and stopping and worst-case OSPs and show that the fixed point solution of the game solves the OSP. Finally, using an approximate dynamic programming method, we develop a simulation-based iterative scheme that computes the optimal controls. The method applies in settings in which neither the system dynamics nor the reward function are known. Hence, the agent need only observe its realised rewards by interacting with the environment. In this paper, we tackled the problem of fault-tolerance within RL in which the controller seeks to obtain a control that is robust against catastrophic failures. To formally characterise the optimal behaviour, we constructed a new discrete-time SG of control and stopping. We established the existence of an equilibrium value then, using a contraction mapping argument, showed that the game can be solved by iterative application of a Bellman operator and constructed an approximate dynamic programming algorithm so that the game can be solved by simulation. Assumption A.2. Ergodicity: i) Any invariant random variable of the state process is P −almost surely (P −a.s.) a constant. Assumption A.3. Markovian transition dynamics: the transition probability function P satisfies the following equality: Assumption A.4. The constituent functions {R, G} in J are square integrable: that is, R, G ∈ L 2 (µ).
The paper tackles fault-tolerance under random and adversarial stoppages.
{"src": "scitldr-AIC_train", "paper_id": "Bygw86VKwS"}