Safely exploring an unknown dynamical system is critical to the deployment of reinforcement learning (RL) in physical systems where failures may have catastrophic consequences. In scenarios where one knows little about the dynamics, diverse transition data covering relevant regions of state-action space is needed to apply either model-based or model-free RL. Motivated by the cooling of Google's data centers, we study how one can safely identify the parameters of a system model with a desired accuracy and confidence level. In particular, we focus on learning an unknown linear system with Gaussian noise assuming only that, initially, a nominal safe action is known. Define safety as satisfying specific linear constraints on the state space (e.g., requirements on process variable) that must hold over the span of an entire trajectory, and given a Probably Approximately Correct (PAC) style bound on the estimation error of model parameters, we show how to compute safe regions of action space by gradually growing a ball around the nominal safe action. One can apply any exploration strategy where actions are chosen from such safe regions. Experiments on a stylized model of data center cooling dynamics show how computing proper safe regions can increase the sample efficiency of safe exploration.
In the context of robust parameter estimation, there is a line of work on bounded noise models in system identification @cite_15 @cite_9 @cite_23 . This paradigm does not assume any statistical form of the noise inherent in state observations, but instead assumes only that the noise is bounded by an unknown constant. Researchers have studied the existence and characteristics of estimators for various optimality concepts @cite_14 . In our setting, we assume a statistical distribution for the noise model, but we compute robust, safe actions with respect to the uncertainty in the model parameters estimated from trajectory data.
@cite_9 @cite_15 @cite_14 @cite_23
2103988321 1966891634 1896002532
A method is discussed for estimating the state of a linear dynamic system using noisy observations, when the input to the dynamic system and the observation errors are completely unknown except for bounds on their magnitude or energy. The state estimate is actually a set in state space rather than a single vector. The optimum estimate is the smallest calculable set which contains the unknown system state, but it is usually impractical to calculate this set. A recursive algorithm is developed which calculates a time-varying ellipsoid in state space that always contains the system's true state. Unfortunately the algorithm is still unproven in the sense that its performance has not yet been evaluated. The algorithm is closely related in structure but not in performance to the algorithm obtained when the system inputs and observation errors are white Gaussian processes. The algorithm development is motivated by the problem of tracking an evasive target, but the results have wider applications. The state equation x_ n = A_ n x_ n-1 +B_ n W_ n and the output equation y_ n = C_ n x_ n +D_ n W_ n relate the finite-dimensional vectors x_ n , y_ n and w n . The initial state x 0 is known to belong to a set X 0 and each w n . to a set W n , where X_ 0 , W_ 1 , . . . , W_ N are compact and convex. Define S_ n (y_ 1 , ... , y_ n ) as the set of all possible values of x n compatible with observation of outputs y_ 1 , ... , y_ n ; it is compact and convex. S n plays the same role as the a posteriori distribution in the stochastic case and is determined recursively by S_ n-1 (y_ 1 , ... , y_ n-1 ) and y n . The sets involved are completely characterized by their support functions. The law of evolution of the support function of S n is established. Some special cases and applications are pointed out. In many problems such as linear and nonlinear regressions, parameter and state estimation of dynamic systems, state-space and time series prediction, interpolation, smoothing and functions approximation, one has to evaluate some unknown variable using available data. The data are always associated with some uncertainty and it is necessary to evaluate how this uncertainty affects the estimated variables. Typically, the problem is approached assuming a probabilistic description of uncertainty and applying statistical estimation theory. An interesting alternative approach, referred to as set membership or unknown but bounded (UBB) error description, has been investigated since the late 1960s. In this approach, uncertainty is described by an additive noise which is known only to have given integral (typically l2 of l1) or componentwise (l∞) bounds. In this paper we review the main results of this theory, with special attention to the most recent advances obtained in the case of componentwise bounds.