In many robotic applications, some aspects of the system dynamics can be modeled accurately while others are difficult to obtain or model. We present a novel reinforcement learning (RL) method for continuous state and action spaces that learns with partial knowledge of the system and without active exploration. It solves linearly-solvable Markov decision processes (L-MDPs), which are well suited for continuous state and action spaces, based on an actor-critic architecture. Compared to previous RL methods for L-MDPs and path integral methods which are model based, the actor-critic learning does not need a model of the uncontrolled dynamics and, importantly, transition noise levels; however, it requires knowing the control dynamics for the problem. We evaluate our method on two synthetic test problems, and one real-world problem in simulation and using real traffic data. Our experiments demonstrate improved learning and policy performance.
As pAC can learn from data containing samples of passive dynamics, it bears resemblance to batch RL methods @cite_0 . A popular and model-free batch RL method is fitted Q-iteration @cite_14 @cite_12 , which finds policy from collected data without a model of system dynamics. It searches for actions that minimize the Q-value, which requires that either the action space be discrete or the Q-function has structure such as being quadratic due to computational cost. In contrast, pAC uses policy that is analytically derived from the estimated Z-value, parameter for transition noise, and known control dynamics.
@cite_0 @cite_14 @cite_12
192920577 2416434549
Batch reinforcement learning is a subfield of dynamic programming-based reinforcement learning. Originally defined as the task of learning the best possible policy from a fixed set of a priori-known transition samples, the (batch) algorithms developed in this field can be easily adapted to the classical online case, where the agent interacts with the environment while learning. Due to the efficient use of collected data and the stability of the learning process, this research area has attracted a lot of attention recently. In this chapter, we introduce the basic principles and the theory behind batch reinforcement learning, describe the most important algorithms, exemplarily discuss ongoing research within this field, and briefly survey real-world applications of batch reinforcement learning. Abstract In this paper a new offline model-free approximate Q-iteration is proposed. Following the idea of Fitted Q-iteration, we use a computational scheme based on Functional Networks, which have been proved to be a powerful alternative to Neural Networks, because they do not require a large number of training samples. We state a condition for the convergence of the proposed technique and we apply it to three classical control problems, namely, a DC motor, a pendulum swing up, a robotic arm. We present a comparative study to show the approximation capabilities of our method with a relatively small number of training samples.