Sparse reward is one of the biggest challenges in reinforcement learning (RL). In this paper, we propose a novel method called Generative Exploration and Exploitation (GENE) to overcome sparse reward. GENE dynamically changes the start state of agent to the generated novel state to encourage the agent to explore the environment or to the generated rewarding state to boost the agent to exploit the received reward signal. GENE relies on no prior knowledge about the environment and can be combined with any RL algorithm, no matter on-policy or off-policy, single-agent or multi-agent. Empirically, we demonstrate that GENE significantly outperforms existing methods in four challenging tasks with only binary rewards indicating whether or not the task is completed, including Maze, Goal Ant, Pushing, and Cooperative Navigation. The ablation studies verify that GENE can adaptively tradeoff between exploration and exploitation as the learning progresses by automatically adjusting the proportion between generated novel states and rewarding states, which is the key for GENE to solving these challenging tasks effectively and efficiently.
, The methods of experience replay focus on which experiences to store and how to use them to speed up the training. Prioritized experience replay @cite_1 measures the priority of experiences by the magnitude of their temporal-difference errors and replays transitions with high priority more frequently. andrychowicz2017hindsight ( andrychowicz2017hindsight ) proposed hindsight experience replay (HER) for goal-based tasks. HER is inspired by that one can learn almost as much from achieving an undesired outcome as from the desired one. After experiencing some episodes, HER arbitrarily selects a set of additional goals and uses them to replace the original goals of the transitions in the reply buffer. However, learning additional goals slows down the learning process and by random exploration the agent rarely sees a real reward signal. Moreover, HER only works on off-policy methods.
@cite_1
2201581102
Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.