We propose a new sample-efficient methodology, called Supervised Policy Update (SPU), for deep reinforcement learning. Starting with data generated by the current policy, SPU formulates and solves a constrained optimization problem in the non-parameterized proximal policy space. Using supervised regression, it then converts the optimal non-parameterized policy to a parameterized policy, from which it draws new samples. The methodology is general in that it applies to both discrete and continuous action spaces, and can handle a wide variety of proximity constraints for the non-parameterized optimization problem. We show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems, and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. The SPU implementation is much simpler than TRPO. In terms of sample efficiency, our extensive experiments show SPU outperforms TRPO in Mujoco simulated robotic tasks and outperforms PPO in Atari video game tasks. The policy gradient problem in deep reinforcement learning (DRL) can be defined as seeking a parameterized policy with high expected reward. An issue with policy gradient methods is poor sample efficiency BID10 BID21 BID27 BID29 . In algorithms such as REINFORCE BID28 , new samples are needed for every gradient step. When generating samples is expensive (such as robotic environments), sample efficiency is of central concern. The sample efficiency of an algorithm is defined to be the number of calls to the environment required to attain a specified performance level BID10 .Thus , given the current policy and a fixed number of trajectories (samples) generated, the goal of the sample efficiency problem is to construct a new policy with the highest performance improvement possible. To do so, it is desirable to limit the search to policies that are close to the original policy π θ k BID21 BID29 BID24 . Intuitively , if the candidate new policy π θ is far from the original policy π θ k , it may not perform better than the original policy because too much emphasis is being placed on the relatively small batch of new data generated by π θ k , and not enough emphasis is being placed on the relatively large amount of data and effort previously used to construct π θ k .This guideline of limiting the search to nearby policies seems reasonable in principle, but requires a distance η(π θ , π θ k ) between the current policy π θ k and the candidate new policy π θ , and then attempt to solve the constrained optimization problem: DISPLAYFORM0 subject to η(π θ , π θ k ) ≤ δwhereĴ(π θ | π θ k , new data) is an estimate of J(π θ ), the performance of policy π θ , based on the previous policy π θ k and the batch of fresh data generated by π θ k . The objective (1) attempts to maximize the performance of the updated policy, and the constraint (2) ensures that the updated policy is not too far from the policy π θ k that was used to generate the data. Several recent papers BID21 BID24 belong to the framework (1)-(2).Our work also strikes the right balance between performance and simplicity. The implementation is only slightly more involved than PPO . Simplicity in RL algorithms has its own merits. This is especially useful when RL algorithms are used to solve problems outside of traditional RL testbeds, which is becoming a trend BID30 BID16 .We propose a new methodology, called Supervised Policy Update (SPU), for this sample efficiency problem. The methodology is general in that it applies to both discrete and continuous action spaces, and can address a wide variety of constraint types for (2). Starting with data generated by the current policy, SPU optimizes over a proximal policy space to find an optimal non-parameterized policy. It then solves a supervised regression problem to convert the non-parameterized policy to a parameterized policy, from which it draws new samples. We develop a general methodology for finding an optimal policy in the non-parameterized policy space, and then illustrate the methodology for three different definitions of proximity. We also show how the Natural Policy Gradient and Trust Region Policy Optimization (NPG/TRPO) problems and the Proximal Policy Optimization (PPO) problem can be addressed by this methodology. While SPU is substantially simpler than NPG/TRPO in terms of mathematics and implementation, our extensive experiments show that SPU is more sample efficient than TRPO in Mujoco simulated robotic tasks and PPO in Atari video game tasks.Off-policy RL algorithms generally achieve better sample efficiency than on-policy algorithms BID8 . However, the performance of an on-policy algorithm can usually be substantially improved by incorporating off-policy training BID17 , BID26 ). Our paper focuses on igniting interests in separating finding the optimal policy into a two-step process: finding the optimal non-parameterized policy, and then parameterizing this optimal policy. We also wanted to deeply understand the on-policy case before adding off-policy training. We thus compare with algorithms operating under the same algorithmic constraints, one of which is being on-policy. We leave the extension to off-policy to future work. We do not claim state-of-the-art results.
first posing and solving the sample efficiency optimization problem in the non-parameterized policy space, and then solving a supervised regression problem to find a parameterized policy that is near the optimal non-parameterized policy.
{"src": "scitldr-AIC_train", "paper_id": "SJxTroR9F7"}