A key strategy in work on exploration in RL has been to include an auxiliary (`intrinsic') reward [1]}, [2]}, [3]}, [4]}, such as for novelty, which encourages the agent to visit unfamiliar states or situations. However, since deep RL generally deals with high-dimensional perceptual observations, it is rare for exactly the same perceptual observation to recur. The question thus arises of how to quantify novelty, and a range of innovative techniques have been proposed to address this problem [5]}, [6]}, [7]}, [8]}. Another approach to intrinsically motivated exploration is to base it not on novelty but on uncertainty, encouraging the agent to enter parts of the environment where its predictions are less confident [9]}. And still other work has pursued the idea of allowing agents to learn or evolve their own intrinsic motivations, based on task experience [10]}, [11]}, [12]}.
