Computer Science Engineering
Computer Science-2021.gz:143397
Computer Science
016534668843e051058efe58806f095364ee12ae
No Need for Interactions: Robust Model-Based Imitation Learning using Neural ODE

Interactions with either environments or expert policies during training are needed for most of the current imitation learning (IL) algorithms. For IL problems with no interactions, a typical approach is Behavior Cloning (BC). However, BC-like methods tend to be affected by distribution shift. To mitigate this problem, we come up with a Robust Model-Based Imitation Learning (RMBIL) framework that casts imitation learning as an end-to-end differentiable nonlinear closed-loop tracking problem. RMBIL applies Neural ODE to learn a precise multi-step dynamics and a robust tracking controller via Nonlinear Dynamics Inversion (NDI) algorithm. Then, the learned NDI controller will be combined with a trajectory generator, a conditional VAE, to imitate an expert's behavior. Theoretical derivation shows that the controller network can approximate an NDI when minimizing the training loss of Neural ODE. Experiments on Mujoco tasks also demonstrate that RMBIL is competitive to the state-of-the-art generative adversarial method (GAIL) and achieves at least 30% performance gain over BC in uneven surfaces.


I. INTRODUCTION
For the majority of the recent Imitation Learning (IL) works, interactions with environments are considered, and several algorithms have been proposed to solve the IL problem in this context, such as Inverse Reinforcement Learning (IRL) [1], [2]. The state-of-the-art Generative Adversarial Imitation Learning (GAIL) [3] is also based on prior IRL works. The success of GAIL gains popularity of the adversarial IL (AIL) framework in IL research field [4], [5], [6], [7]. However, the reinforcement loop inside AIL method has a risk of driving the learned policy to visit unsafe or undefined states-space during training. As such, in this work, we attend the scenario where the imitated policy is NOT allowed to interact with environments or access information from the expert policy in training phase.
For the scenarios where interactions are not accessible, a common approach is still the Behavior Cloning [8], [9], which imitates the expert by approximating the conditional distribution of actions over states in a supervised learning fashion. With sufficient demonstrations collected from the expert, the BC methods have successfully found its wide applications in autonomous driving [10] and robot locomotion [11]. Nevertheless, the robustness of BC is not guaranteed because of the compounding errors caused by covariate shift issue [12].
Some efforts have been made to address the compounding errors issue under BC framework. Ross and Bagnell proposed DAgger [12] that enables the expert policy to correct the behaviors of imitated policy during training. Mahler and Goldberg [13] introduced Disturbances for Augmenting Robot Trajectory (DART) that applies the expert policy to generate sub-optimal demonstrations. Torabi et al. presented BCO [11] with an inverse dynamics model for inferring actions from observations through environment exploration. Nevertheless, all these algorithms require interactions with either the environment or the expert policy during training, which is against our problem settings.
In order to effectively utilize the embedded physical information from the finite expert demonstrations, we choose the model-based scheme rather than BC like or model-free approaches. However, performance of model-based approach is directly influenced by accuracy of the learned system dynamics, as well as stability and robustness of learned controller. In the deep learning field, such properties are hard to be verified and guaranteed. Therefore, we borrow concepts and definitions from the nonlinear control theories so as to analyze the proposed framework.
As depicted by Osa et al. [14], the model-based imitation learning (MBIL) problem could be considered as a typical closed-loop tracking control problem, which is composed of system dynamics, tracking controller, and trajectory generator (shown in Fig.1-(a)). In traditional nonlinear control field, such tracking control problem can be transformed into an equivalent linear system through Nonlinear Dynamics Inversion (NDI) [15], [16]) algorithm (shown in Fig.1-(b)). As a result, the transformed linear system can be flexibly controlled by a linear controller. By applying NDI concept to MBIL problem, we could adopt some nonlinear control methodology, such as stability and robustness analysis [16]. However, the stability of NDI is guaranteed if the dynamics model matches the physical system [17].
The recent Neural ODE research [18] presents a new framework to learn a precise multi-steps continuous system dynamics with irregular time-series data. With this seminal theory, Rubanova et al. [19] and Zhong et al. [20] further proposed some improvements in their works. As one key part of this work, we advocate a Neural ODE based purely data-driven approach for learning a multi-steps continuous actuated dynamics by applying the zero-order-hold to the control inputs. Based on the precise learned dynamics, we propose the Robust Model-Based Imitation Learning (RMBIL) framework, which formulates MBIL as an endto-end differentiable nonlinear tracking control problem via NDI algorithm, as illustrated in Fig.1-(c), where the system dynamics and NDI control policy are trained with a closedloop Neural ODE model. On the other hand, the trajectory generator block in MBIL methodology is equivalent to the Reference Model (RM) in NDI framework. The RM module is usually a hand-designed kinodynamics trajectory approximator which mimics the expert's behaviors via a high-order polynomial function. Inspired by recent works in trajectories predictions [21], [22], [23], we further adopt the Conditional Variational Autoencoder (CVAE) [24], [25], conditioned on the previous state, to replace the hand-designed RM block. At inference, only the learned NDI controller and the decoder from the trained CVAE are used for imitating expert's trajectories, as shown in Fig.1-(d). In addition, to increase robustness of the proposed framework, we take the concept of the sliding surface from the Sliding Mode Control (SMC), which is a traditional nonlinear robust control algorithm [26]. Empirically, we find that the sliding surface could be approximated by injecting noise to the system state during the training of the NDI controller with a closed-loop Neural ODE model. As a result, RMBIL obtains an approximated 30% performance increase over BC, and is competitive to GAIL approach, in the disturbed environments.
II. PRELIMINARY Given expert demonstrations is a finite sequence of stateaction pairs for T ∈ N samples, x k e,ti is sampled from state trajectory x(t) ∈ R n , u k e,ti is sampled from policy outputs u(t) ∈ R m , and s k is a context vector that represents the initial state x k (0) of sequence k. A common method to solve MBIL is to explicitly train a discrete forward dynamics: , as well as a tracking controller: u k ti = π φ (x k r,ti+1 , x k ti ), where the reference state x k r,ti is provided by a trajectory generator: x k r,ti+1 = π ψ (x k ti ), for all N tasks. Since all models should be independent of s k , without losing the generality, we assume N = 1 for the following derivation in order to remove superscript k for simplicity.

A. Nominal Control: Nonlinear Dynamics Inversion
The main concept behind NDI is to cancel nonlinearities of a system via input-output linearization [26]. To review the theory of NDI, consider a continuous input-affine system defined as: where y(t) ∈ R k is the output vector, a : R n → R n and h : R n → R k are smooth vector fields, and G ∈ R n×m is a matrix whose columns are smooth vector fileds. To find the explicit relation between the outputs y(t) and the control inputs u(t), Eq.
(2) will be differentiated repeatedly (for simplicity, we denote (3) is not zero, then the input-output relation is found. To achieve desired outputs y des , Eq.(3) is reformulated as: where ν =ẏ des is a virtual input that is tracked by the derivative of outputẏ. Hence, by controlling ν, the corresponding u ndi in Eq.(4) would take y in Eq.(2) to the desired output y des through Eq.(1). A common approach to obtain suitable ν is using a linear proportional feedback controller: where K n is a k×k diagonal matrix with a manually chosen gain value. Eq.
is a non-square matrix, then the pseudo-inverse method may be applied. In addition, we assume our system is fullyobservable and y = x, which are conventions under MDP assumption in IL field [3], [27], [11]. Therefore, ∇h(x) is an identity matrix, and Eq.(4) could be rewritten as follows: where ν(t) is a function of y des (t) and y(t) according to Eq.(5). By applying Eq.(6) to the physical platform, the whole system becomes linear and can be flexibly controlled by Eq.(5). However, stability of a NDI controller is guaranteed if the dynamics model (Eq. (1) and (2)) matches the physical platform [17]. Yellow block is a neural network for computing the time derivative of actuated dynamics. Green block is a continuous function u zoh (t) approximated by ZOH method based on discrete input sequence ue,t i . Although the internal state x(t) is continuous, the predicted outputs are discrete with regard to the specified time sequence t 0:τ . Hence, the loss can be computed based on both discrete trajectories.

B. Ancillary Control: Sliding Mode Control
The idea behind SMC is to design a nonlinear controller to force the disturbed system to slide along the desired trajectory. As defined in [28], the SMC scheme consists of two steps: (1) Find a sliding surface S such that the system will stay confined within it when the system trajectory reaches the surface within the finite time. In our case S = x des (t), where x des (t) = y des (t) for the fully-observable system. (2) Design a switching function σ(x(t)), which represents the distance between the current state and sliding surface, for the feedback control law u rn to lead the system trajectory to intersect and stay on the S. In this work, we The sufficient condition [28] for the global stability of SMC is σ (x(t))σ(x(t)) < 0. To satisfy the condition and dynamics model (Eq.(1)), the resulting nonlinear robust control policy u rn is designed as follows (Denote x(t) = x): where K s is a positive definite n × n matrix. Therefore, by adding an additional feedback term u smc , which is dependent on the switching function σ(x), to the nominal NDI controller u ndi , we may obtain a robust NDI control law u rn . In addition, Eq.(9) also implies that u rn would be identical to u ndi when x err = 0.

III. METHODOLOGY A. Multi-steps Actuated Dynamics using Neural ODE
Instead of learning a discrete dynamics, we are interested in a continuous-time dynamics with control inputs that can be formulated as:ẋ(t) = f θ (x(t), u(t)). To approximate such differential dynamical systems with a neural network f θ , we adopt the Neural ODE model proposed by Chen et al. [18], which solves the initial-value problem (Eq.(10)) in a supervised learning fashion by backpropagating through a black-box ODE solver using adjoint sensitivity method [29] for memory efficiency.
x t0 , . . . ,x tn = ODESolver(f θ , x t0 , (t 0 , . . . , t n )) (10) To update the dynamics with respect to weights θ, the loss function L θ can be constructed as a L2-norm between the predicted statex ti and the true state x e,ti ∈ D for a certain time-horizon τ , that is: L θ (τ ) = x e,ti:τ −x ti:τ 2 . However, Eq.(10) can only handle an autonomous systeṁ x(t) =f θ (x(t)). In order to apply Neural ODE to an actuated dynamics, Rubanova et al. [19] proposed a RNN based framework to map the system into latent space and solve the latent autonomous dynamics, while Zhong et al. [20] introduced an augmented dynamics by appending the control input to the system state under a strong assumption that the control signal must stay constant over the time-horizon τ .
In this work, we propose a more intuitive and general method to handle the actuated dynamics. We construct a continuous input function u zoh (t) based on the sampled control signal u e,ti ∈ D by zero-order-hold (ZOH). For each internal integration time t s inside Neural ODE, the actuated dynamics could always obtain the corresponding control signal by accessing the input function u e,ts ← u zoh (t = t s ), as illustrated in Fig. 2. As such, the Neural ODE (Eq.(10)) could be directly applied to an arbitrary dynamic system with external controls without any additional assumption or constraint. Once L θ (τ ) → 0 with τ ∆t, where ∆t is the integration step of ODE solver, we obtain a precise multisteps continuous actuated dynamicsx(t) =f θ (x(t), u(t)) based on discrete expert demonstrations D. Unfortunately, compared to the baseline model such as multilayer perceptron (MLP), the integration solver inside Neural ODE becomes the bottleneck of inference time, which limits the learned dynamics to be integrated into classic modelbased planning and control scheme such as Model Predictive Control (MPC). [30], [31], [32].

B. Learning based NDI Controller via Neural ODE
To overcome the above inference time bottleneck caused by the ODE solver, we train a controller network via the learned dynamics with the Neural ODE (Eq.(10)). As a result, only the learned controller will be adopted for controlling the physical system. Based on this concept, the Proposition 3.1 expresses how a control policy, parameterized by φ, approximates the NDI formulation (Eq.(6)).

Proposition 3.1: NDI Controller Training
Proposition 3.1 implies that by minimizing the loss L φ (τ ), the NDI controller is learned with Neural ODE if the trained dynamicsf θ is accurate and can be expressed as an affine system. To prove this proposition, we start from the loss function with respect to the controller weight φ.
The first term at RHS of Eq.(11) is zero since x t0 is given as an initial condition. We let τ=1 without loss of generality, and the results can be easily extended to multi-steps case. Fig. 3. Performance comparison of proposed MBIL and RMBIL versus baselines with respect to the number of demonstrations. The x-axis is the number of expert demonstration used in the training, and y-axis is the normalized rewards (expert as one and random as zero). The shadow area beside each line represents its variance calculated based on 50 test trajectories. Note: GAIL needs environments interactions during training, while RMBIL does not.
The training goal of Neural ODE, which belongs to the supervised learning family, is to minimize the loss L φ . According to Eq.(13), the loss will be the minimum if the first term equals to (or approximates) the second term as follows: x In order to obtain the NDI formulation from Eq. (14), we introduce the first assumption that the equal sign still holds after applying the time derivative on both sides of Eq. (14) if L θ (τ ) ≈ 0 along trajectories, where τ ∆t. In general, for arbitrary two functions whose value are identical at a certain point, the values of time derivative at the same point are not guaranteed to be equal. However, in Eq. (14), the state-action pairs' sequence data ξ T used on both sides is from the same demonstrations D. Therefore, if the training loss L θ of the learned dynamics approaches zero with τ ∆t (in practice, we stop the training as L θ < , where < 0.01), then both sides of Eq.(14) represent the same system trajectory x(t). Hence, we could apply the time derivative operator to Eq. (14) and the equal sign is still held as follows: To further derive Eq.(16), we introduce the second assumption that the true dynamics, where the proposed RMBIL is trying to mimic, is an input-affine system. This assumption is roughly satisfied with most physical controllable platforms. Under this assumption, Eq.(16) can be reformulated as: where ν(t 0 ) = ν(x(t 0 ), x r (t 1 )) from Eq.(5). By substituting the controller networkπ φ for the expected control inputŝ u(t), the relation to the NDI formulation emerges from Eq. (18). Since the two assumptions discussed above may not be fulfilled perfectly in practice, we replace the equal sign with an approximation sign. In addition, we reformulate Eq. (18) to bringπ φ to the LHS:π where Eq. (20) is derived from replacing the RHS of Eq. (19) with Eq. (6). As a result, with Eq. (14), (15) and (20), Proposition 3.1 is validated. We have to emphasize here that u ndi (t) is only applied to the state space where we observed from the dataset D. Although the proposition indicates that a learned policy with Neural ODE can be treated as a NDI controller, the robustness of the controller is not guaranteed [17].

C. Robustness Improvement through Noise Injection
To improve robustness of the learned NDI controllerπ θ , we borrow the concept of SMC discussed in Section II-B. Rather than building a hierarchical ancillary controller, we intend to design a robust NDI policy networkπ φ which is end-to-end differentiable through Neural ODE. Inspired by DART [13], we refine the trained NDI controllerπ φ by adding zero-mean gaussian noise to the internal statex(t) within Neural ODE so as to construct the switching function σ (Eq. (7)). As stated with proposition 3.2, the robustness of the refined controllerπ φ would be improved because an ancillary SMC policy (Eq.(9)) is formulated automatically when L φ (the training loss forπ φ ) approaching zero. Due to space limits, please refer to the link 1 for the proof details.

Proposition 3.2: Controller Robustness Improvement
Refine the learned controllerπ φ with noised injected state, sampled from Gaussian distribution,

D. Conditional VAE for Trajectory Generation
The training objective for the trajectory generator in MBIL framework is to predict the reference state given the past states. To support multi-task scenario, inspired by [22], [27] , we use CVAE [24] to predict the future trajectoryx r (t), but without the need for embedding multi-steps dynamics information because the learnable dynamicsf θ and robust controllerπ φ have carried on such information. At training, the generative mode of the proposed CVAE q ψe (z|x ti , c), parameterized by ψ e , encodes the current state x ti to an embedding vector z conditioned on the variable c, which in Algorithm 1 Dynamics and Controller Training Input: dataset D, σ x for noise injection, τ for solver horizon, and r as convergence indicator. // Multi-steps dynamics training 1: Initialize model parameters θ and φ randomly 2: Bypassing the controller model π φ 3: while L θ (τ ) > do 4: Predict trajectoriesx ti:τ using Eq.(10) 5: Compute loss L θ (τ ) = x e,ti:τ −x ti:τ Refine π φ based on L φ (τ ) = x e,ti:τ −x ti:τ 2 17: end while our case is the previous state x ti−1 . Given z, the inference network p ψ d (x ti |z, c), prarmeterized by ψ d , decodes the current state x ti under the same condition variable c = x ti−1 . Both network parameters (ψ e and ψ d ) are updated by minimizing the following loss: where D KL is the Kullback-Leibler divergence between the approximated posterior and conditional prior p(z|c) which is assumed as the standard Gaussian distribution N (0, I). In addition, the first term on the RHS of Eq.(21) represents the reconstruction loss. For inference, by feeding the current system state x(t i ) as the context vector c, the trained inference network p ψ d (x ti+1 |z, x ti ) would predict the state at next time step x(t i+1 ) which can be treated as the reference x r (t i+1 ) for the tracking controllerπ φ , where the embedding vector z is sampled from the assumed prior N (0, I).

E. Procedure for Training and Inference
The training pipeline is composed of two main modules, the Neural ODE based dynamics-controller module and the CVAE based generator module. As described in Algorithm.1, the dynamics-controller module is trained in three phases with Neural ODE model. (1) Starting by training the dynamics modelf θ . (2) Once L θ (τ ) < , enable the control policy networkπ φ to learn a NDI controller. (3) When the loss L φ (τ ) < , start adding noise to the internal statex(t) to obtain a robust controllerπ φ . The method to apply the noise injection inside Neural ODE is illustrated in Fig.1-(c). In contrast, the training procedure of the generator module is straightforward, that is, minimizing the loss L ψ defined in Eq.(21) until both reconstruction loss and KL divergence converge. For inference, as illustrated in Fig.1-(d), only the trained robust tracking controllerπ φ and the trained inference network p ψ d are used for driving the physical platform to mimic expert's behavior given initial state x 0 .

IV. EXPERIMENTS A. Environment Setup
We choose Hopper, Walker2d and HalfCheetah from Ope-nAI Gym [33] under Mujoco [34] physics engine as the simulation environments. To collect demonstrations, we use TRPO [35] algorithm via stable-baselines [36] to train the expert policies. For each environment, we record the stateaction pair sequence ξ k T with T = 1000 steps for k = 50 episodes under random initial state .

B. Baselines
We compare RMBIL against four baselines: BC, DART, GAIL and MBIL, where MBIL is a non-robust version of RMBIL (without noise injection). For BC method, a vanilla MLP is implemented to imitate the expert policy. For DART and GAIL methods, we slightly modify the official implementation to fit with our dataset format. For a fair comparison, the network size of the imitated policy for all methods are identical. The details of hyperparameters and network structure are listed in Table.I .

C. Performance Comparison
Given expert demonstrations, BC, MBIL, and RMBIL can be trained directly. For DART, we allow the algorithm to access the expert policies to generate sub-optimal trajectories during training. For GAIL, the training is unstable and requires interaction with the environment. Therefore, we train GAIL with 10 4 interactions while saving the checkpoint every 10 3 interactions, then choose the best one for comparison. At inference, for each method, we execute the trained policy for 50 episodes under random initial conditions defined by the environment, then compute the average and standard deviation. The normalized rewards against the number of demonstrations are shown in Fig. 3 by treating the performance of expert policy as one and random policy as zero. We can observe that RMBIL outperforms BC and DART in the Fig. 4. Robustness evaluation for RMBIL (with different feedback gains) versus baselines. The y-axis is the normalized rewards. For each method, we plot average (the marker), standard deviation (solid error bar) and minimum-maximum range (dash error bar) with respect to 50 test episodes. The number behind RMBIL stands for the value of the feedback gain used at inference. The number behind GAIL stands for the number of environment interactions used during training (x1000). Note: GAIL needs continuous environments interactions during training, while RMBIL does not. case of extremely few demonstrations. Higher performance at low-data regimes implies that our method has better sample efficiency. With enough number of demonstrations, RMBIL can achieve similar performance to the expert policy, same as DART and BC. In contrast, the performance of GAIL is not directly dependent on demonstrations amount. In addition, there is a performance gap between MBIL and RMBIL in high-data regimes (8% for Hopper, 10% for Walker2d). This phenomenon supports our discussion in Sec. III-C., namely, the trained dynamics is not perfect, therefore, the learned controller should consider robustness in order to handle the model inaccuracies.

D. Robustness evaluation
In order to evaluate the robustness across different trained policies, we set up two types of environment disturbances for Hopper and Walker2D cases as described in TABLE.II. Because we are interested in how the feedback gain K n of the linear controller inside RMBIL affects the robustness of the learned policy, we train RMBIL with K n = 0.1 based on 50 demonstrations. At inference, we compare the performance of learned robust NDI controller under different gain (0.1,1 and 10). The normalized average reward over 50 trajectories in Fig.4 shows that RMBIL with high-gain obtains better mean than the ones with low-gain in both cases. This observation meets the linear control theory that robustness of the transformed linear system can be enhanced by increasing the feedback gain [37]. In addition, compared to DART, RMBIL has higher rewards mean in Hopper case and similar performance in Walker2d case. In contrast, the comparison of GAIL is hard to analyze since the performance of its imitated policy varying significantly with respect to the number of environments interactions. However, we could still roughly observe that the mean and variance of RMBIL is located on the average performance range of GAIL in both Walker2d and Hopper cases.
On the other hand, through Walker2d cases in Fig.4, we could observe the covariate shift issue exists in the BC method, where the trained BC policy achieves the same rewards as the expert in the default environment, however, when encountering unknown disturbances, the performance of BC policy degrades dramatically. In comparison, since the proposed RMBIL is based on a precise multi-steps dynamics and a nonlinear controller with noise injection, the learned robust controller could avoid overfitting to the expert demonstrations and overcome the environment uncertainties at testing time.
V. CONCLUSION In this work, we presented RMBIL, a Neural ODE based approach for imitation learning without the need for access to expert policy or environment interaction during training. To the best of our knowledge, we are the first to study IL problem from the perspective of traditional nonlinear control theory with both theoretical and empirical supports. With the theoretical analysis, we prove that the learnable control network inside Neural ODE could approximate an NDI controller by minimizing the training loss. Experiments on complicated Mujco tasks show that RMBIL can achieve the same performance as the expert policy. In addition, for unstable systems, such as Hopper and Waker2d, with environmental disturbances, the performance of RMBIL is competitive to GAIL algorithm and outperforms BC method. Future works may incorporate other existing classic nonlinear control theories and explore multi-tasks applications.