Recently, model-free reinforcement learning algorithms have been shown to solve challenging problems by learning from extensive interaction with the environment. A significant issue with transferring this success to the robotics domain is that interaction with the real world is costly, but training on limited experience is prone to overfitting. We present a method for learning to navigate, to a fixed goal and in a known environment, on a mobile robot. The robot leverages an interactive world model built from a single traversal of the environment, a pre-trained visual feature encoder, and stochastic environmental augmentation, to demonstrate successful zero-shot transfer under real-world environmental variations without fine-tuning.
In model-free RL, the policy is implemented by a function approximator, commonly a deep neural network. The system is usually trained either to map observations and actions to the expected future reward (as in Q-learning) or to directly approximate the probability distribution over actions that maximizes expected reward (as in policy search). When using a deep neural network as a function approximator, a policy is learned from experience with the environment, usually by gradient descent on trajectories of @math -tuples. In value-based methods such as Q-learning, the policy consists of using a value estimate (the @math function) to choose actions with the highest return, whereas in policy search methods, the policy is parameterized directly and a value function is used to update the parameters of the policy in the direction of positive outcomes. We primarily consider a value-based method known as @cite_21 .
@cite_21
2280163991
Efficient exploration in complex environments remains a major challenge for reinforcement learning. We propose bootstrapped DQN, a simple algorithm that explores in a computationally and statistically efficient manner through use of randomized value functions. Unlike dithering strategies such as epsilon-greedy exploration, bootstrapped DQN carries out temporally-extended (or deep) exploration; this can lead to exponentially faster learning. We demonstrate these benefits in complex stochastic MDPs and in the large-scale Arcade Learning Environment. Bootstrapped DQN substantially improves learning times and performance across most Atari games.