Actor-critic methods solve reinforcement learning problems by updating a parameterized policy known as an actor in a direction that increases an estimate of the expected return known as a critic. However, existing actor-critic methods only use values or gradients of the critic to update the policy parameter. In this paper, we propose a novel actor-critic method called the guide actor-critic (GAC). GAC firstly learns a guide actor that locally maximizes the critic and then it updates the policy parameter based on the guide actor by supervised learning. Our main theoretical contributions are two folds. First, we show that GAC updates the guide actor by performing second-order optimization in the action space where the curvature matrix is based on the Hessians of the critic. Second, we show that the deterministic policy gradient method is a special case of GAC when the Hessians are ignored. Through experiments, we show that our method is a promising reinforcement learning method for continuous controls.
 The goal of reinforcement learning (RL) is to learn an optimal policy that lets an agent achieve the maximum cumulative rewards known as the return BID31 . Reinforcement learning has been shown to be effective in solving challenging artificial intelligence tasks such as playing games BID20 and controlling robots BID6 .Reinforcement learning methods can be classified into three categories: value-based, policy-based, and actor-critic methods. Value-based methods learn an optimal policy by firstly learning a value function that estimates the expected return. Then, they infer an optimal policy by choosing an action that maximizes the learned value function. Choosing an action in this way requires solving a maximization problem which is not trivial for continuous controls. While extensions to continuous controls were considered recently, they are restrictive since specific structures of the value function are assumed BID10 BID3 .On the other hand, policy-based methods, also called policy search methods BID6 , learn a parameterized policy maximizing a sample approximation of the expected return without learning the value function. For instance, policy gradient methods such as REIN-FORCE BID34 use gradient ascent to update the policy parameter so that the probability of observing high sample returns increases. Compared with value-based methods, policy search methods are simpler and naturally applicable to continuous problems. Moreover, the sample return is an unbiased estimator of the expected return and methods such as policy gradients are guaranteed to converge to a locally optimal policy under standard regularity conditions BID32 . However, sample returns usually have high variance and this makes such policy search methods converge too slowly.Actor-critic methods combine the advantages of value-based and policy search methods. In these methods, the parameterized policy is called an actor and the learned value-function is called a critic.The goal of these methods is to learn an actor that maximizes the critic. Since the critic is a low variance estimator of the expected return, these methods often converge much faster than policy search methods. Prominent examples of these methods are actor-critic BID32 BID15 , natural actor-critic BID25 , trust-region policy optimization BID27 , and asynchronous advantage actor-critic BID21 . While their approaches to learn the actor are different , they share a common property that they only use the value of the critic, i.e., the zero-th order information, and ignore higher-order ones such as gradients and Hessians w.r.t. actions of the critic 1 . To the best of our knowledge, the only actor-critic methods that use gradients of the critic to update the actor are deterministic policy gradients (DPG) BID29 and stochastic value gradients . However, these two methods do not utilize the second-order information of the critic.In this paper, we argue that the second-order information of the critic is useful and should not be ignored. A motivating example can be seen by comparing gradient ascent to the Newton method: the Newton method which also uses the Hessian converges to a local optimum in a fewer iterations when compared to gradient ascent which only uses the gradient BID24 . This suggests that the Hessian of the critic can accelerate actor learning which leads to higher data efficiency. However, the computational complexity of second-order methods is at least quadratic in terms of the number of optimization variables. For this reason, applying second-order methods to optimize the parameterized actor directly is prohibitively expensive and impractical for deep reinforcement learning which represents the actor by deep neural networks.Our contribution in this paper is a novel actor-critic method for continuous controls which we call guide actor-critic (GAC). Unlike existing methods, the actor update of GAC utilizes the secondorder information of the critic in a computationally efficient manner. This is achieved by separating actor learning into two steps. In the first step, we learn a non-parameterized Gaussian actor that locally maximizes the critic under a Kullback-Leibler (KL) divergence constraint. Then, the Gaussian actor is used as a guide for learning a parameterized actor by supervised learning. Our analysis shows that learning the mean of the Gaussian actor is equivalent to performing a second-order update in the action space where the curvature matrix is given by Hessians of the critic and the step-size is controlled by the KL constraint. Furthermore, we establish a connection between GAC and DPG where we show that DPG is a special case of GAC when the Hessians and KL constraint are ignored. Actor-critic methods are appealing for real-world problems due to their good data efficiency and learning speed. However, existing actor-critic methods do not use second-order information of the critic. In this paper, we established a novel framework that distinguishes itself from existing work by utilizing Hessians of the critic for actor learning. Within this framework, we proposed a practical method that uses Gauss-Newton approximation instead of the Hessians. We showed through experiments that our method is promising and thus the framework should be further investigated.Our analysis showed that the proposed method is closely related to deterministic policy gradients (DPG). However, DPG was also shown to be a limiting case of the stochastic policy gradients when the policy variance approaches zero BID29 . It is currently unknown whether our framework has a connection to the stochastic policy gradients as well, and finding such a connection is our future work.Our main goal in this paper was to provide a new actor-critic framework and we do not claim that our method achieves the state-of-the-art performance. However, its performance can still be improved in many directions. For instance, we may impose a KL constraint for a parameterized actor to improve its stability, similarly to TRPO BID27 . We can also apply more efficient policy evaluation methods such as Retrace BID22 ) to achieve better critic learning.
This paper proposes a novel actor-critic method that uses Hessians of a critic to update an actor.
{"src": "scitldr-AIC_train", "paper_id": "BJk59JZ0b"}