The transfer of reinforcement learning (RL) techniques into real-world applications is challenged by safety requirements in the presence of physical limitations. Most RL methods, in particular the most popular algorithms, do not support explicit consideration of state and input constraints. In this paper, we address this problem for nonlinear systems with continuous state and input spaces by introducing a predictive safety filter, which is able to turn a constrained dynamical system into an unconstrained safe system, to which any RL algorithm can be applied out-of-the-box'. The predictive safety filter receives the proposed learning input and decides, based on the current system state, if it can be safely applied to the real system, or if it has to be modified otherwise. Safety is thereby established by a continuously updated safety policy, which is based on a model predictive control formulation using a data-driven system model and considering state and input dependent uncertainties.
Gaussian process (GP) regression has been widely used in supervised machine learning due to its flexibility and inherent ability to describe uncertainty in function estimation. In the context of control, it is seeing increasing use for modeling of nonlinear dynamical systems from data, as it allows the direct assessment of residual model uncertainty. We present a model predictive control (MPC) approach that integrates a nominal system with an additive nonlinear part of the dynamics modeled as a GP. Approximation techniques for propagating the state distribution are reviewed and we describe a principled way of formulating the chance constrained MPC problem, which takes into account residual uncertainties provided by the GP model to enable cautious control. Using additional approximations for efficient computation, we finally demonstrate the approach in a simulation example, as well as in a hardware implementation for autonomous racing of remote controlled race cars, highlighting improvements with regard to both performance and safety over a nominal controller. A novel adaptive output feedback control technique for uncertain linear systems is proposed, able to cope with input and output constraints and measurement noise. At each time step, the collected input-output data are exploited to refine the set of models that are consistent with the available information on the system. Then, the control input is computed according to a receding horizon strategy, which guarantees recursive constraint satisfaction for all the admissible models, hence also for the actual plant. The technique relies only on the solution of linear and quadratic programs. The effectiveness of the approach is illustrated in a numerical example. Abstract A robust model predictive control (RMPC) approach for linear systems with bounded state-dependent uncertainties is proposed. Such uncertainties can arise from unmodeled non-linearities or external disturbances, for example. By explicitly considering the state dependency of the uncertainty sets in the RMPC approach, it is shown how closed-loop performance can be improved over existing approaches that consider worst-case uncertainty. Being able to handle state-dependent uncertainties is particularly relevant in learning-based MPC where the system model is learned from data and confidence in the model typically varies over the state space. The efficacy of the proposed approach for learning-based RMPC is illustrated with a numerical example, where uncertainty sets are obtained from data using Gaussian Process regression. Spanish MINECO Grant PRX15-00300 and projects DPI2013-48243-C2-2-R and DPI2016-76493-C3-1-R. UK Engineering and Physical Research Council, grant no.EP J012300 1. Advances in the direct computation of Lyapunov functions using convex optimization make it possible to efficiently evaluate regions of attraction for smooth non-linear systems. Here we present a feedback motion-planning algorithm which uses rigorously computed stability regions to build a sparse tree of LQR-stabilized trajectories. The region of attraction of this non-linear feedback policy âprobabilistically coversâ the entire controllable subset of state space, verifying that all initial conditions that are capable of reaching the goal will reach the goal. We numerically investigate the properties of this systematic non-linear feedback design algorithm on simple non-linear systems, prove the property of probabilistic coverage, and discuss extensions and implementation details of the basic algorithm. Trial-and-error based reinforcement learning (RL) has seen rapid advancements in recent times, especially with the advent of deep neural networks. However, the majority of autonomous RL algorithms require a large number of interactions with the environment. A large number of interactions may be impractical in many real-world applications, such as robotics, and many practical systems have to obey limitations in the form of state space or control constraints. To reduce the number of system interactions while simultaneously handling constraints, we propose a model-based RL framework based on probabilistic Model Predictive Control (MPC). In particular, we propose to learn a probabilistic transition model using Gaussian Processes (GPs) to incorporate model uncertainty into long-term predictions, thereby, reducing the impact of model errors. We then use MPC to find a control sequence that minimises the expected long-term cost. We provide theoretical guarantees for first-order optimality in the GP-based transition models with deterministic approximate inference for long-term planning. We demonstrate that our approach does not only achieve state-of-the-art data efficiency, but also is a principled way for RL in constrained environments. In this paper, we present a novel constraint tightening approach for nonlinear robust model predictive control (MPC). This approach uses a simple constructive constraint tightening based on growing tubes. Contrary to other approaches, we require no complex offline computations to obtain a stabilizing control law. Instead, we consider the notion of incremental stabilizability and design tubes based on an estimate of the achievable exponential decay rate. In addition, we show how this tightening can be used as an ad-hoc modification to improve the robustness of MPC without terminal constraints. We study the system theoretic properties of the resulting closed-loop system, including bounds on the region of attraction and the minimal robust positively invariant (RPI) set. Within an MPC framework without terminal constraints, the proposed constraint tightening leads to a nonlinear robust controller without complex design procedures, which makes it appealing for practical applications.  Reinforcement learning has been successfully used to solve difficult tasks in complex unknown environments. However, these methods typically do not provide any safety guarantees during the learning process. This is particularly problematic, since reinforcement learning agent actively explore their environment. This prevents their use in safety-critical, real-world applications. In this paper, we present a learning-based model predictive control scheme that provides high-probability safety guarantees throughout the learning process. Based on a reliable statistical model, we construct provably accurate confidence intervals on predicted trajectories. Unlike previous approaches, we allow for input-dependent uncertainties. Based on these reliable predictions, we guarantee that trajectories satisfy safety constraints. Moreover, we use a terminal set constraint to recursively guarantee the existence of safe control actions at every iteration. We evaluate the resulting algorithm to safely explore the dynamics of an inverted pendulum and to solve a reinforcement learning task on a cart-pole system with safety constraints.