In model-based reinforcement learning, the agent interleaves between model learning and planning.   These two components are  inextricably intertwined. If the model is not able to provide sensible long-term prediction, the executed planer would exploit model flaws, which can yield catastrophic failures. This paper focuses on building a model that reasons about the long-term future and demonstrates how to use this for efficient planning and exploration. To this end, we build a latent-variable autoregressive model by leveraging recent ideas in variational inference. We argue that forcing latent variables to carry future information through an auxiliary task substantially improves long-term predictions. Moreover, by planning in the latent space, the planner's solution is ensured to be within regions where the model is valid. An exploration strategy can be devised by searching for unlikely trajectories under the model. Our methods achieves higher reward faster compared to baselines on a variety of tasks and environments in both the imitation learning and model-based reinforcement learning settings. Reinforcement Learning (RL) is an agent-oriented learning paradigm concerned with learning by interacting with an uncertain environment. Combined with deep neural networks as function approximators, deep reinforcement learning (deep RL) algorithms recently allowed us to tackle highly complex tasks. Despite recent success in a variety of challenging environment such as Atari games BID4 and the game of Go , it is still difficult to apply RL approaches in domains with high dimensional observation-action space and complex dynamics.Furthermore, most popular RL algorithms are model-free as they directly learn a value function BID34 or policy BID43 ) without trying to model or predict the environment's dynamics. Model-free RL techniques often require large amounts of training data and can be expensive, dangerous or impossibly slow, especially for agents and robots acting in the real world. On the other hand, model-based RL BID49 BID14 BID11 provides an alternative approach by learning an explicit representation of the underlying environment dynamics. The principal component of model-based methods is to use an estimated model as an internal simulator for planning, hence limiting the need for interaction with the environment. Unfortunately, when the dynamics are complex, it is not trivial to learn models that are accurate enough to later ensure stable and fast learning of a good policy.The most widely used techniques for model learning are based on one-step prediction. Specifically, given an observation o t and an action a t at time t, a model is trained to predict the conditional distribution over the immediate next observation o t+1 , i.e p(o t+1 | o t , a t ). Although computationally easy, the one-step prediction error is an inadequate proxy for the downstream performance of model-based methods as it does not account for how the model behaves when com-posed with itself. In fact, one-step modelling errors can compound after multiple steps and can degrade the policy learning. This is referred to as the compounding error phenomenon BID51 BID0 BID54 . Other examples of models are autoregressive models such as recurrent neural networks BID32 that factorize naturally as log p θ (o t+1 , a t+1 , o t+2 , a t+2 , . . . | o t , a t ) = t log p θ (o t+1 , a t+1 | o 1 , a 1 , . . . o t , a t ). Training autoregressive models using maximum likelihood results in 'teacher-forcing' that breaks the training over one-step decisions. Such sequential models are known to suffer from accumulating errors as observed in BID30 .Our key motivation is the following -a model of the environment should reason about (i.e. be trained to predict) long-term transition dynamics p θ (o t+1 , a t+1 , o t+2 , a t+2 , . . . | o t , a t ) and not just single step transitions p θ (o t+1 | o t , a t ). That is, the model should predict what will happen in the long-term future, and not just the immediate future. We hypothesize (and test) that such a model would exhibit less cascading of errors and would learn better feature embeddings for improved performance.One way to capture long-term transition dynamics is to use latent variables recurrent networks. Ideally, latent variables could capture higher level structures in the data and help to reason about long-term transition dynamics. However, in practice it is difficult for latent variables to capture higher level representation in the presence of a strong autoregressive model as shown in BID17 BID16 ; BID18 . To overcome this difficulty , we leverage recent advances in variational inference. In particular, we make use of the recently proposed Z-forcing idea BID16 , which uses an auxiliary cost on the latent variable to predict the long-term future. Keeping in mind that more accurate long-term prediction is better for planning, we use two ways to inject future information into latent variables. Firstly, we augment the dynamics model with a backward recurrent network (RNN) such that the approximate posterior of latent variables depends on the summary of future information. Secondly, we force latent variables to predict a summary of the future using an auxiliary cost that acts as a regularizer. Unlike one-step prediction, our approach encourages the predicted future observations to remain grounded in the real observations.Injection of information about the future can also help in planning as it can be seen as injecting a plan for the future. In stochastic environment dynamics, unfolding the dynamics model may lead to unlikely trajectories due to errors compounding at each step during rollouts.In this work, we make the following key contributions:1. We demonstrate that having an auxiliary loss to predict the longer-term future helps in faster imitation learning. 2. We demonstrate that incorporating the latent plan into dynamics model can be used for planning (for example Model Predictive Control) efficiently. We show the performance of the proposed method as compared to existing state of the art RL methods. 3. We empirically observe that using the proposed auxiliary loss could help in finding sub-goals in the partially observable 2D environment. In this work we considered the challenge of model learning in model-based RL. We showed how to train, from raw high-dimensional observations, a latent-variable model that is robust to compounding error. The key insight in our approach involve forcing our latent variables to account for long-term future information. We explain how we use the model for efficient planning and exploration. Through experiments in various tasks, we demonstrate the benefits of such a model to provide sensible long-term predictions and therefore outperform baseline methods. Mujoco Tasks We evaluate on 2 Mujoco tasks BID52 , the Reacher and the Half Cheetah task BID52 . The Reacher tasks is an object manipulation task consist of manipulating a 7-DoF robotic arm to reach the goal, the agent is rewarded for the number of objects it reaches within a fixed number of steps. The HalfCheetah task is continuous control task where the agent is awarded for the distance the robots moves.For both tasks, the experts are trained using Trust Region Policy Optimization (TRPO) BID43 . We generate 10k expert trajectories for training the student model, all models are trained for 50 epochs. For the HalfCheetah task, we chunk the trajectory (1000 timesteps) into 4 chunks of length 250 to save computation time.Car Racing task The Car Racing task BID28 ) is a continuous control task where each episode contains randomly generated trials. The agent (car) is rewarded for visiting as many tiles as possible in the least amount of time possible. The expert is trained using methods in BID19 . We generate 10k trajectories from the expert. For trajectories of length over 1000, we take the first 1000 steps. Similarly to Section 5.1, we chunk the 1000 steps trajectory into 4 chunks of 250 for computation purposes.BabyAI The BabyAI environment is a POMDP 2D Minigrid envorinment BID10 with multiple tasks. For our experiments, we use the PickupUnlock task consistent of 2 rooms, a key, an object to pick up and a door in between the rooms. The agent starts off in the left room where it needs to find a key, it then needs to take the key to the door to unlock the next room, after which, the agent will move into the next room and find the object that it needs to pick up. The rooms can be of different sizes and the difficulty increases as the size of the room increases. We train all our models on room of size 15. It is not trivial to train up a reinforcement learning expert on the PickupUnlock task on room size of 15. We use curriculum learning with PPO BID44 for training our experts. We start with a room size of 6 and increase the room size by 2 at each level of curriculum learning.We train the LSTM baseline and our model both using imitation learning. The training data are 10k trajectories generated from the expert model. We evaluate the both baseline and our model every 100 iterations on the real test environment (BabyAI environment) and we report the reward per episode. Experiments are run 5 times with different random seeds and we report the average of the 5 runs.Wheeled locomotion We use the Wheeled locomotion with sparse rewards environment from (CoReyes et al., 2018) . The robot is presented with multiple goals and must move sequentially in order to reach each reward. The agent obtains a reward for every 3 goal it reaches and hence this is a task with sparse rewards. We follow similar setup to BID13 , the number of explored trajectories for MPC is 2048, MPC re-plans at every 19 steps. However, different from (Co-Reyes et al., 2018), we sample latent variables from our sequential prior which depends on the summary of the past events h t . This is in comparison to BID13 , where the prior of the latent variables are fixed. Experiments are run 3 times and average of the 3 runs are reported.
incorporating, in the model, latent variables that encode future content improves the long-term prediction accuracy, which is critical for better planning in model-based RL.
{"src": "scitldr-AIC_train", "paper_id": "SkgQBn0cF7"}