The other approach of solving hard exploration problems is a sample-efficient use of experiences [1]}, [2]}. Prioritized experience replay (PER) method introduced in Schaul et al. [2]} samples experiences that can give more information to the RL agent and thus have more `surprisal' - defined by the temporal-difference (TD) error - in higher probability. In a similar sense, self-imitation learning (SIL) introduced in Oh et al. [4]} samples only the `good' experiences where the actual reward is larger than the agent's value estimate (i.e., estimate from the value function or Q function). However, prioritized sampling based on the agent's value estimate is susceptible to outliers and spikes, which may lead to destabilizing the agent itself [5]}.
Moreover, our explorative algorithm aims to preserve sufficient diversity and find many possible solutions instead of finding the most efficient route to a single solution. Thus, we modify the formulation of priority using the estimates for sample novelty.
<FIGURE>