In the third IMGEP phase, we need to get back to the proposed goal. One variant of Go-Explore simply resets the agent to the desired goal, but this requires an environment which can be set to an arbitrary state. Instead, we follow the more generic `policy-based Go-Explore' approach, which uses a goal-conditioned policy network to get back to a goal. The concept of goal-conditioning, which was introduced by [1]}, is also the common approach in IMGEP approaches. A well-known addition to goal-based RL approaches is hindsight experience replay [2]}, which allows the agent to make more efficient use of the data through counterfactual goal relabelling. Although Go-Explore did not use hindsight in their work, we do include it as an extension in our approach.
