State Entropy Maximization with Random Encoders for Efficient Exploration
Recent exploration methods have proven to be a recipe for improving sample efficiency in deep reinforcement learning (RL) However, efficient exploration in high dimensional observation spaces still remains a challenge. This paper presents Random Encoders for Efficient Exploration (RE3) an exploration method that utilizes state entropy as an intrinsic reward. In order to estimate state entropy in environments with high dimensional observations, we utilize a k nearest neighbor entropy estimator in the low dimensional representation space of a convolutional encoder. In particular, we find that the state entropy can be estimated in a stable and compute efficient manner by utilizing a randomly initialized encoder, which is fixed throughout training. Our experiments show that RE3 significantly improves the sample efficiency of both model free and model based RL methods on locomotion and navigation tasks from DeepMind Control Suite and MiniGrid benchmarks. We also show that RE3 allows learning diverse behaviors without extrinsic rewards, effectively improving sample efficiency in downstream tasks. Source code and videos are available at https: /sites.google.com/view/re3 rl.
ICML