Improving DDPG via Prioritized Experience Replay
Experience replay plays an important role in reinforcement learning. It reuses previous experiences to prevent the input data from being highly correlated. Recently, a deep reinforcement learning algorithm with experience replay, called deep deterministic policy gradient (DDPG) has achieved good performance in many continuous control tasks. However, it assumes the experiences are of equal importance and samples them uniformly, which obviously neglects the difference in the value of each individual experience. To improve the efficiency of experience replay in DDPG method, we propose to replace the original uniform experience replay with prioritized experience replay. We test the algorithms in five tasks in the OpenAI Gym, a testbed for reinforcement learning algorithms. In the experiment, we find that DDPG with prioritized experience replay mechanism significantly outperforms that with uniform sampling in terms of training time, training stability and final performance. We also find that our algorithm can achieve better performance in three tasks compared with some state of the art algorithms like Q prop, which directly proves the effectiveness of our proposed scheme. Our weekly report and codes are available at https:/github.com/cardwing/Codes for RL Project.