In this work, we propose a novel formulation of planning which views it as a probabilistic inference problem over future optimal trajectories. This enables us to use sampling methods, and thus, tackle planning in continuous domains using a fixed computational budget.    We design a new algorithm,  Sequential Monte Carlo Planning, by leveraging classical methods in Sequential Monte Carlo and Bayesian smoothing in the context of control as inference. Furthermore, we show that Sequential Monte Carlo Planning can capture multimodal policies and can quickly learn continuous control tasks. To exhibit intelligent behaviour machine learning agents must be able to learn quickly, predict the consequences of their actions, and explain how they will react in a given situation. These abilities are best achieved when the agent efficiently uses a model of the world to plan future actions. To date, planning algorithms have yielded very impressive results. For instance, Alpha Go BID36 relied on Monte Carlo Tree Search (MCTS) BID23 ) to achieve super human performances. Cross entropy methods (CEM) BID34 have enabled robots to perform complex nonprehensile manipulations BID11 and algorithms to play successfully Tetris BID39 . In addition, iterative linear quadratic regulator (iLQR) BID21 BID20 BID41 enabled humanoid robots tasks to get up from an arbitrary seated pose .Despite these successes, these algorithms make strong underlying assumptions about the environment. First, MCTS requires a discrete setting, limiting most of its successes to discrete games with known dynamics. Second , CEM assumes the distribution over future trajectories to be Gaussian, i.e. unimodal. Third , iLQR assumes that the dynamics are locally linear-Gaussian, which is a strong assumption on the dynamics and would also assume the distribution over future optimal trajectories to be Gaussian. For these reasons, planning remains an open problem in environments with continuous actions and complex dynamics. In this paper, we address the limitations of the aforementioned planning algorithms by creating a more general view of planning that can leverage advances in deep learning (DL) and probabilistic inference methods. This allows us to approximate arbitrary complicated distributions over trajectories with non-linear dynamics.We frame planning as density estimation problem over optimal future trajectories in the context of control as inference BID10 BID45 BID43 Rawlik et al., 2010; BID47 BID31 . This perspective allows us to make use of tools from the inference research community and, as previously mentioned, model any distribution over future trajectories. The planning distribution is complex since trajectories consist of an intertwined sequence of states and actions. Sequential Monte Carlo (SMC) BID38 BID13 BID27 methods are flexible and efficient to model such a T −1 t≥1 p env (s t+1 |s t , a t ) T t≥1 π θ (a t |s t ) denotes the probability of a trajectory x 1:T under policy π θ . FIG8 .1: O t is an observed optimality variable with probability p(O t |s t , a t ) = exp(r(s t , a t )).x t = (s t , a t ) are the state-action pair variables considered here as latent.Traditionally, in reinforcement learning (RL) problems, the goal is to find the optimal policy that maximizes the expected return E q θ [ T t=1 γ t r t ]. However, it is useful to frame RL as an inference problem within a probabilistic graphical framework BID33 BID45 BID30 . First, we introduce an auxiliary binary random variable O t denoting the "optimality" of a pair (s t , a t ) at time t and define its probability 1 as p(O t = 1|s t , a t ) = exp(r(s t , a t )). O is a convenience variable only here for the sake of modeling. By considering the variables (s t , a t ) as latent and O t as observed, we can construct a Hidden Markov Model (HMM) as depicted in figure 2.1. Notice that the link s → a is not present in figure 2.1 as the dependency of the optimal action on the state depends on the future observations. In this graphical model , the optimal policy is expressed as p(a t |s t , O t:T ).The posterior probability of this graphical model can be written as 2 : DISPLAYFORM0 r(s t , a t ) + log p(a t ) .(2.1)It appears clearly that finding optimal trajectories is equivalent to finding plausible trajectories yielding a high return.1 as in BID30 , if the rewards are bounded above, we can always remove a constant so that the probability is well defined.2 Notice that in the rest of the paper, we will abusively remove the product of the action priors T t=1 p(at) = exp T t=1 log p(at) from the joint as in BID30 . We typically consider this term either constant or already included in the reward function. See Appendix A.2 for details.Many control as inference methods can be seen as approximating the density by optimizing its variational lower bound: BID43 . Instead of directly differentiating the variational lower bound for the whole trajectory, it is possible to take a message passing approach such as the one used in Soft Actor-Critic (SAC) BID17 and directly estimate the optimal policy p(a t |s t , O t:T ) using the backward message, i.e a soft Q function instead of the Monte Carlo return. DISPLAYFORM1 In this work, we have introduced a connection between planning and inference and showed how we can exploit advances in deep learning and probabilistic inference to design a new efficient and theoretically grounded planning algorithm. We additionally proposed a natural way to combine model-free and model-based reinforcement learning for planning based on the SMC perspective. We empirically demonstrated that our method achieves state of the art results on Mujoco. Our result suggest that planning can lead to faster learning in control tasks.However, our particle-based inference method suffers some several shortcomings. First, we need many particles to build a good approximation of the posterior, and this can be computationally expensive since it requires to perform a forward pass of the policy, the value function and the model for every particle. Second, resampling can also have adverse effects, for instance all the particles could be resampled on the most likely particle, leading to a particle degeneracy. More advanced SMC methods dealing with this issue such as backward simulation BID32 or Particle Gibbs with Ancestor Sampling (PGAS) (Lindsten et al., 2014) have been proposed and using them would certainly improve our results.Another issue we did not tackle in our work is the use of models of the environment learned from data. Imperfect model are known to result in compounding errors for prediction over long sequences. We chose to re-plan at each time step (Model Predictive Control) as it is often done in control to be more robust to model errors. More powerful models or uncertainty modeling techniques can also be used to improve the accuracy of our planning algorithm. While the inference and modeling techniques used here could be improved in multiple ways, SMCP achieved impressive learning speed on complex control tasks. The planning as inference framework proposed in this work is general and could serve as a stepping stone for further work combining probabilistic inference and deep reinforcement learning.
Leveraging control as inference and Sequential Monte Carlo methods, we proposed a probabilistic planning algorithm.
{"src": "scitldr-AIC_train", "paper_id": "ByetGn0cYX"}