A Receding Horizon Approach for Simultaneous Active Learning and Control using Gaussian Processes

This paper proposes a receding horizon active learning and control problem for dynamical systems in which Gaussian Processes (GPs) are utilized to model the system dynamics. The active learning objective in the optimization problem is presented by the exact conditional differential entropy of GP predictions at multiple steps ahead, which is equivalent to the log determinant of the GP posterior covariance matrix. The resulting non-convex and complex optimization problem is solved by the Sequential Convex Programming algorithm that exploits the first-order approximations of non-convex functions. Simulation results of an autonomous racing car example verify that using the proposed method can significantly improve data quality for model learning while solving time is highly promising for real-time applications.

Abstract-This paper proposes a receding horizon active learning and control problem for dynamical systems in which Gaussian Processes (GPs) are utilized to model the system dynamics. The active learning objective in the optimization problem is presented by the exact conditional differential entropy of GP predictions at multiple steps ahead, which is equivalent to the log determinant of the GP posterior covariance matrix. The resulting non-convex and complex optimization problem is solved by the Sequential Convex Programming algorithm that exploits the first-order approximations of nonconvex functions. Simulation results of an autonomous racing car example verify that using the proposed method can significantly improve data quality for model learning while solving time is highly promising for real-time applications.

I. INRODUCTION
Modeling the system dynamics play a pivotal role in the performance of model-based control techniques such as Receding Horizon Control (RHC, also known as Model Predictive Control). Nevertheless, for many complex dynamical systems, obtained mathematical models are often insufficiently accurate due to the existence of uncertainties and ignored dynamical parts. This challenge motivates learning-based models for control, which leverages Machine Learning (ML) techniques to model dynamical systems from data and certain prior knowledge. For example, Gaussian Processes (GPs) [1] have been applied for dynamics and control recently [2]. A fundamental but challenging problems in learning the system dynamics using GPs is how to obtain a training dataset such that the learned models can efficiently capture the actual dynamics. This is because, in most control applications, the experiments for data collection are limited by time, cost, or constraints of environment, whereas using only historical data is not suitable due to the lack of input excitation. For instance, consider a motivating example of an autonomous racing car. In this example, the experiments are constrained by narrow and sharp racing tracks, while the historical data obtained from manual control or simple automatic control techniques do not have sufficient excitation. A better approach would be simultaneous learning and control where the car explores the state space for learning as quickly as possible while satisfying other control objectives such as maintaining safety and tracking a racing trajectory. The learning objective requires that the system dynamics are sampled at the states associated with informative GP inputs. Given current models, the goal can be achieved by driving the system to the state where the information from collected data can minimize prediction uncertainty at the region of interest. Once the system is controlled to the new state, the GP inputs and outputs are collected, then the model is retrained. This repetitive procedure is referred to as active learning for dynamical systems.
The general active learning framework has been applied to various domains (see [3] for a survey) to address the problem pertaining to optimal training data collection. However, the active learning problem for dynamical systems has only been studied recently [4]- [7]. In [4], a greedy scheme for choosing the next data point in Optimal Experiment Design (OED) was proposed by exploiting the maximum variance and information gain methods. The models learned from the experiment was then used for an RHC problem. Meanwhile, active learning using multi-step look-ahead was considered [5]- [7] and shows better performance than single-step approach. Particularly, in [5], based on the conditional differential entropy of multi-step GP predictions, the authors first determined the most informative data point within the region of interest by a greedy algorithm, then steering the system towards that state using RHC. The paper [6] considered an RHC problem for dual control (i.e., simultaneous learning and control) in which the objective function consists of a control objective and a knowledge-gain (information-gain) objective for learning the dynamics. The knowledge-gain objective was obtained using concepts from information theory, such as mutual information or relative entropy. In [7], an RHC problem for OED including both active learning objective represented by the differential entropy of GP predictions and dynamic constraints was formulated. However, to limit the computational burden, only an upper or lower bound of the differential entropy in [7] and the estimated knowledge gain in [6] of the multi-step GP predictions was utilized. That is, the information-gain metrics for multi-step GP predictions were replaced by the sum of individual metrics for step-wise predictions over the horizon. As a result, the problems can be solved in continuous domain by nonlinear programming solvers instead of grid-based methods. However, computation time for solving the optimization problems was not reported in those papers to judge whether the used methods are suitable for real-time control.
In this paper, we presents a Receding Horizon Active Learning and Control (RHALC) problem for dynamical systems using the GP regressions. The presented problem formulation covers both the dual control problem in [6] and the problem for experiment design in [7]. However, instead of approximate information-gain metrics, we take the exact conditional differential entropy of multi-step GP predictions into account. The resulting optimization problem involving the GP dynamics and the log determinants of posterior covariance matrices is thus non-convex and highly complex that may prevent the success of any non-linear programming solvers. To overcome this challenge, we apply the Sequential Convex Programming (SCP) method [8] to efficiently address the problem by performing the first-order approximations of the GP means and the GP posterior covariance matrices. The effectiveness of the proposed method is validated by simulations of an autonomous racing car example. The results on trajectory tracking control performance of the racing car and prediction accuracy of the learned models show that the active learning can improve data quality for model learning in both offline learning (experiment design) and simultaneous learning and control. In addition, the reported fast computation time of the SCP algorithm demonstrates its capability for real-time implementation.
The remainder of this paper is organized as follows. The RHALC problem formulation is introduced in Section II. Section III provides SCP algorithm, while the simulation results are reported and discussed in Section IV. Finally, Section V concludes the paper.

II. PROBLEM FORMULATION
In this section, we introduce a general Receding Horizon Active Learning and Control (RHALC) formulation in which the GPs are employed to model the system dynamics while the exact conditional differential entropy is used as an optimization metric in active learning.

A. Gaussian Process Regression
Consider a latent function f : R n → R and N noisy observations y (i) of it, y (i) = f x (i) + ǫ (i) , at inputs x (i) ∈ R n and with i.i.d. Gaussian noise ǫ (i) ∼ N 0, σ 2 n , for i = 1, . . . , N . We will use X = [x (1) , . . . , x (N ) ] ∈ R n×N to denote the collection of all input vectors and Y = [y (1) , . . . , y (N ) ] ∈ R N to denote the collection of the corresponding observed outputs. Let D = (X, Y) be the set of observation data of f . A GP of f , which will be denoted by G f , is a probability distribution over all possible realizations of f and can be formally defined as a collection of random variables, any finite number of which have a joint Gaussian distribution [1]. It is fully specified by a covariance function k(x, x ′ ; θ) and a mean function m(x; θ). The mean function is employed to include prior knowledge about the unknown function. In this paper, the mean function is assumed to be zero without loss of generality. At , in which the predictive mean vector µ ⋆ and the M ×M posterior covariance matrix Σ ⋆ are computed as follows where σ 2 n is the Gaussian noise variance, I is an identity matrix of appropriate dimensions, K ⋆⋆ ∈ R M×M is the covariance matrix at x ⋆ , K ⋆ ∈ R M×N is the cross-covariance matrix between x ⋆ and X, K ∈ R N ×N is the covariance matrix at X, in which the elements K ij of each matrix is computed by K ij = k(x (i) , x (j) ). Note that in this paper, we only utilize the GP means without uncertainty propagation to represent the predicted values of the nonlinear dynamics. For more details on GPs and its usage in controls, readers are referred to [1], [2].

B. Receding Horizon Active Learning and Control with Gaussian Process
We define the control input vector as u ∈ R nu , the vector of GP output variables y ∈ R ny and the vector of non-GP variables z ∈ R nz . For any variable , where is y, z, or u, let k denote its value at time step k. The state of the system at time step k comprises y k and z k . We denote the GP dynamic model as y k ∼ G(x k ). The input vector x k , or features, of the GP is formed from current and past values of the control inputs u τ and non-GP states z τ , for τ ≤ k, as well as from past GP outputs y τ , for τ < k. To simplify the notation and formulation, we assume that y is scalar; however, our results can be readily extended to multivariate y. Given an input x k , letȳ k = µ(x k ) denote the predictive mean of the GP model G(·) at x k .
Let H > 0 be the length of the control horizon, t be the current time step and I t = {t, . . . , t+H −1} be the set of all time steps in the control horizon at time step t. DenoteȲ t = {ȳ k |k ∈ I t }, Z t = {z k |k ∈ I t }, U t = {u k |k ∈ I t }, and X t = {x k |k ∈ I t } as the sets collecting the predictive GP output means, the non-GP states, the control inputs, and GP inputs over the control horizon. To simplify the mathematical notations, we will use [X] to denote the vector concatenation of all vectors in a set X (e.g., . Given a GP model trained on the data generated up to the current time step, the most informative GP regressor vectors in the next horizon X t can be determined by maximizing the conditional differential entropy of GP predictions at these vectors that is computed by the log determinant of the covariance matrix [9], leading to the following optimization problem where Σ([X t ]) is a H × H posterior covariance matrix of GP predictions at H input vectors in the set X t and can be computed by (1b). The RHALC problem is thus formulated as follows where H(X t ) = log det Σ([X t ]) is an active learning term and J(Ȳ t , U t , Z t ) is an control objective function, g j (Ȳ t , U t , Z t ) ≤ 0 and h j (Ȳ t , U t , Z t ) = 0 are inequality and equality constraints while J ieq and J eq are the sets of inequality and equality constraint indices, respectively. In the problem (3), γ is a positive constant representing a tradeoff between learning and control objectives. Similar to [10], we make the following assumption about the problem (3).
Assumption 1: Suppose that J is convex, each g j is convex, and each h i is affine in the optimization variables U t and Z t . In other words, the non-convexity of the problem (3) results from the GP dynamics and the log determinant of the GP predictive covariance matrix.

III. SEQUENTIAL CONVEX PROGRAMMING FOR RECEDING HORIZON ACTIVE LEARNING AND CONTROL
The problem (3) is highly nonconvex due to the active learning objective and the GP dynamics. Moreover, the complexity of the objective function involving the log determinant of the GP predictive covariance matrix makes the problem (3) computationally intractable. In this section, we employ the Successive Convex Programming (SCP) approach [8] to effectively address this problem.
Suppose that nominal feasible control inputs are given in We then simulate the GP model G(·) over the RHC horizon to obtain the nominal output means can be obtained from these values. Consider small perturbations to the nominal control inputs u k = u ⋆ k + ∆u k , which are collected in ∆U t = {∆u k | k ∈ I t }. They will cause perturbations to the predictive output means and regressor vectors during the MPC horizon as: Using these perturbation variables, the RHALC (3) can be approximated locally around the nominal values by replacing the GP predictive meansȳ k = µ(x k ) and the log determinant of GP predictive covariance matrix with their first order approximations as follows.
Defineỹ k as the first-order approximation of y k around the nominal solution x ⋆ k , which can be computed as follows where from (1a) we have being the gradient of k with respect to the first argument. We also defineỸ t as a collection of y k for k ∈ I t . Meanwhile, the first-order approximation of Note that the derivative of the log determinant of the GP predictive covariance matrix with respect to each element ν j of a vector ν can be computed by With a slight abuse of notations, we will write U t = U ⋆ t + ∆U t . We now obtain the convexified RHALC problem, as stated below.
Constraints (6c) specify a trust region in which the local convexified subproblem is valid. To avoid the artificial infeasibility [8] of the problem due to the approximations (4) and (5), the inequality and equality constraints in (6) are encoded into the objective function by the exact penalty functions [8] leading to the following penalized convex problem minimize ∆Ut,Zt where τ j , ∀j ∈ J ieq and λ j , ∀j ∈ J eq are large positive penalty weights. Under Assumption 1, the penalized subproblem (7) is convex and can be solved efficiently by convex solvers. The SCP algorithm for solving the RHALC problem is outlined in Algorithm 1. For further details on the SCP method and its application to the RHC problem using the GPs, the readers are referred to [8] and [10], respectively.

IV. SIMULATION
This section validates the advantages of the RHALC problem in two scenarios, experiment design and simultaneous learning and control problem, as well as the effectiveness of the SCP algorithm in solving the problems in real-time by a numerical simulation of an autonomous racing car example.

A. Autonomous racing car example
We revisit the example of an autonomous racing car mentioned in Section I. The simulation consists of two phases: learning phase and racing phase. During the learning phase, starting with initial GP models trained on a few historical data, the controller collects and adds new data points and retrain the GP models. Once sufficient data for learning accurate models have been obtained, the learning phase is disabled and the obtained models are utilized to perform tracking control task in the racing phase. Form convex subproblem (7) by using (4) and (5) 5:

14:
else 15: Similar to [11], the kinematic bicycle model of the vehicle is used in the simulation while the following discrete-time dynamics with a sampling time ∆T > 0 are utilized to design the control problem where (x, y) is the position vector of the vehicle, θ is the heading angle, v is the speed of the vehicle, and a and α are respectively the linear acceleration and steering angle of the vehicle. The continuous-time kinematic bicycle model and parameters of the racing car can be found in [11]. The nonlinear components in (8) are learned by three GP models: ∆x k ∼ G x (x p,k ), ∆y k ∼ G y (x p,k ), and ∆θ k ∼ G θ (x a,k ), in which the vectors of GP inputs The GP models result in the following GP dynamical equations The RHALC formulation for this example is given by subject to (8) and ( where the constraints hold for all k ∈ I t , (10c) are velocity bound constraints, (10d) are bound constraints on the control actions, and (10e) are affine constraints representing collision avoidance to the border of the racing track. The objective of this problem consists of two parts: trajectory tracking control and active learning for dynamics. The control objective function J is given by where r k+1 denotes the reference at time step k + 1. Given a vector ν and a positive semidefinite matrix M, we define ν 2 M = ν T Mν. The active learning goals for the GP models are captured by where x p,t+1:t+H and x p,t+1:t+H denotes the concatenated vector of GP inputs from time t + 1 to t + H. To avoid the collision with the track borders, we employed a scheme presented in [12] that limit the movement of the car at each time step in the horizon to lie within two parallel halfplanes. As a result, the collision avoidance scheme can be represented as a set of linear inequality constraints in (10e). All the penalty weights in (7) are chosen to be 10 6 . The parameters of the SCP algorithm are: ρ (0) = 0.1, β fail = 0.5, β succ = 2.0, r 0 = 0.01, r 1 = 0.1, r 2 = 0.3, j max = 100, ǫ = 10 −4 . Two virtual racing tracks developed by the University of California, Berkeley (UCB) [13] and the University of Pennsylvania (UPenn) [14] are used in the simulations.
In what follows, we present two scenarios for the racing car example considered in this paper: Offline learning and Simultaneous Learning and Control (Online Learning).
1) Offline learning: Assume that a large area without obstacles is available for experiments. We design an experiment to obtain an optimal dataset for training the GP dynamics prior to the race. This procedure is also referred to as Optimal Experiment Design (OED) [7]. In the OED problem, only the active learning term is included in the objective function while the control objective J is removed. The constraints (10e) are simplified to bound constraints on x and y to ensure that the car lies within the experimental space.
We compare the optimal experiment design using RHALC with a randomized experiment design where random control inputs that satisfy all the input constraints are applied to the system. In the optimal experiment design, three simple GP models with 25 initial data points are utilized, then the RHALC problem is applied in 50 time steps to collect new data points. The models learned from experimental data are used to perform a reference tracking control task. The receding horizon reference tracking problem can be derived by removing the active learning term in (10).
2) Simultaneous Learning and Control: In the application where a free area for experiments is not available, it is required to perform Simultaneous Active Learning and Control, or Online Learning, in the learning phase. The vehicle is expected to efficiently collect online data to update the GP dynamics while following the racing track and avoiding collision to the borders. Hence, this scenario covers the dual control problem in [6].
To validate the benefits of active learning, we consider two simulations depending on the effect of the active learning term. In the first simulation, the active learning objective is involved to whereas in the second simulation, the active learning is disabled and the vehicle only tracks to the reference and collects online data along the racing track. In this scenrio, three initial GP models trained on 50 initial data points are adopted to control the system at the beginning, then 100 new data samples are collected online. The simultaneous learning and control simulation is conducted using the UPenn track, while the UCB track is utilized for a testing race given the model learned from the previous race.

B. Results and discussions
The trajectories of the autonomous racing car in the offline learning simulation are shown in Fig. 1. Using the GP models from the optimal experiment, the receding horizon tracking controller can accurately track the reference and prevent the collision to the borders. In contrast, the GP models from the randomized experiment apparently are not accurate since the car cannot complete one racing lap without crashing into the borders. Particularly, the car initially can track well to the reference but collision to the borders happens when the car needs to turn sharply. Meanwhile, Fig. 2 shows the trajectories of the car in the Simultaneous Learning and Control simulation. As can be seen from the figure, with the active learning, the vehicle initially fluctuates around the reference to explore the informative states, hence the car does not perfectly track the reference in the first 100 time instants but the safety condition is guaranteed. However, once completing the learning phase, the obtained GP models are accurate so that the vehicle is able to track the reference trajectory in the rest of the racing track and in a new testing track. Meanwhile, without active learning, at the beginning of the race where the racing track is relatively simple, then the car can track to the reference. However, since the learned models do not have enough excitation, the car deviates to the reference trajectory when the track becomes more complicated leading to the crashes to the borders.
Furthermore, the prediction accuracy of the GP models obtained from all simulations are shown in Table I. We compare 4 types of GP models, OE, RE, AL and Non-AL, which are correspondingly obtained from the offline optimal and randomized experiments, and the learning and control simulations with and without active learning. Two validation metrics including the root mean squared errors (RMSEs), and the maximum absolute errors (MAEs) are considered. These validation metrics are computed using the GP predictions on a grid of GP inputs in which 20 linearly spaced points in the region of interest for each input and the corresponding latent non-linear functions. According to the table, with the same number of training data points, three GP models generated from the optimal experiment show better performance in prediction accuracy than those from the randomized experiment. Likewise, based on the metrics for AL and non-AL models, it is obvious that active learning can improve GP precision in simultaneous learning and control.
Regarding the computation time, in the optimal experiment, SCP algorithm takes 0.069 s on average for each time instant, while in the race, it takes 0.041 s and 0.040 s in UCB and UPenn racing tracks, respectively. Meanwhile, in online learning simulation that includes active leaning, SCP  algorithm averagely takes 0.104 s, whereas in control phase, it takes 0.058 s for each time step. Note that all simulations in this work are performed on a DELL computer with a 3.0 GHz Intel Core i5 CPU and 8 Gb RAM, and the Julia programming language is used for the implementation.

V. CONCLUSION
In this paper, a receding horizon active learning and control problem for dynamical systems using Gaussian Processes (GPs) was considered. We first developed a problem formulation subjecting to the GP dynamics while the exact conditional differential entropy was employed as a metric for active learning. The resulting complex and non-convex problem was solved by the Sequential Convex Programming algorithm. The proposed method was validated by numerical simulations of an autonomous racing car example. Not only guarantee the tracking performance in both offline and online learning scenarios, the control algorithm but also can be executed in a reasonable amount of time, which promises its potential practicality for real-time implementation.