In many robotic applications, some aspects of the system dynamics can be modeled accurately while others are difficult to obtain or model. We present a novel reinforcement learning (RL) method for continuous state and action spaces that learns with partial knowledge of the system and without active exploration. It solves linearly-solvable Markov decision processes (L-MDPs), which are well suited for continuous state and action spaces, based on an actor-critic architecture. Compared to previous RL methods for L-MDPs and path integral methods which are model based, the actor-critic learning does not need a model of the uncontrolled dynamics and, importantly, transition noise levels; however, it requires knowing the control dynamics for the problem. We evaluate our method on two synthetic test problems, and one real-world problem in simulation and using real traffic data. Our experiments demonstrate improved learning and policy performance.
Previous approaches for solving L-MDPs are predominantly model based @cite_13 @cite_15 @cite_16 . These efficiently optimize control policies by solving the linearized Bellman in discrete- or continuous-state L-MDPs when the system dynamics is fully known. Our method relaxes this requirement using samples of passive dynamics, while knowing the control dynamics. We also introduce multi-layer neural networks for approximating the value functions in L-MDPs in addition to the previously used radial basis functions.
@cite_15 @cite_16 @cite_13
2128152413 2015383196
We have identified a general class of nonlinear stochastic optimal control problems which can be reduced to computing the principal eigenfunction of a linear operator. Here we develop function approximation methods exploiting this inherent linearity. First we discretize the time axis in a novel way, yielding an integral operator that approximates not only our control problems but also more general elliptic PDEs. The eigenfunction problem is then approximated with a finite-dimensional eigenvector problem - by discretizing the state space, or by projecting on a set of adaptive bases evaluated at a set of collocation states. Solving the resulting eigenvector problem is faster than applying policy or value iteration. The bases are adapted via Levenberg-Marquardt minimization with guaranteed convergence. The collocation set can also be adapted so as to focus the approximation on a region of interest. Numerical results on test problems are provided. Abstract A general class of stochastic optimal control problems has recently been reduced to computing the principle eigenfunction of a linear operator. Here we present an approximation framework for solving such problems by using soft state aggregation over a continuous space. This approach enables us to avoid matrix factorization and take advantage of sparsity by using efficient iterative solvers. Adaptive schemes for basis placement are developed so as to provide higher resolution at the regions of state space that are visited more often. Numerical results on test problems are provided.