Dealing with Sparse Rewards in Continuous Control Robotics via Heavy-Tailed Policies
In this paper, we present a novel Heavy-Tailed Stochastic Policy Gradient
(HT-PSG) algorithm to deal with the challenges of sparse rewards in continuous
control problems. Sparse reward is common in continuous control robotics tasks
such as manipulation and navigation, and makes the learning problem hard due to
non-trivial estimation of value functions over the state space. This demands
either reward shaping or expert demonstrations for the sparse reward
environment. However, obtaining high-quality demonstrations is quite expensive
and sometimes even impossible. We propose a heavy-tailed policy parametrization
along with a modified momentum-based policy gradient tracking scheme (HT-SPG)
to induce a stable exploratory behavior to the algorithm. The proposed
algorithm does not require access to expert demonstrations. We test the
performance of HT-SPG on various benchmark tasks of continuous control with
sparse rewards such as 1D Mario, Pathological Mountain Car, Sparse Pendulum in
OpenAI Gym, and Sparse MuJoCo environments (Hopper-v2). We show consistent
performance improvement across all tasks in terms of high average cumulative
reward. HT-SPG also demonstrates improved convergence speed with minimum
samples, thereby emphasizing the sample efficiency of our proposed algorithm.