We present research using the latest reinforcement learning algorithm for end-to-end driving without any mediated perception (object recognition, scene understanding). The newly proposed reward and learning strategies lead together to faster convergence and more robust driving using only RGB image from a forward facing camera. An Asynchronous Actor Critic (A3C) framework is used to learn the car control in a physically and graphically realistic rally game, with the agents evolving simultaneously on tracks with a variety of road structures (turns, hills), graphics (seasons, location) and physics (road adherence). A thorough evaluation is conducted and generalization is proven on unseen tracks and using legal speed limits. Open loop tests on real sequences of images show some domain adaption capability of our method.
An alternative for unsupervised (or self-supervised) learning, is deep Reinforcement Learning (RL) as it uses reward to train a network to find the most favorable state. The two major interests for RL are that: a) it doesn't require ground truth data and b) the reward can be sparse and delayed which opens new horizons. Indeed, judging a driving control decision at the frame level is complex and deep RL allows rewarding or penalizing after a sequence of decisions. To avoid getting caught in local optima, experience replay memory can be used as in @cite_7 . Another solution is asynchronous learning - coined A3C - as proposed by and successfully applied for learning ATARI games @cite_13 using score as a reward. A3C achieves experience decorrelation with multiple agents evolving in different environments at the same time. @cite_13 they also applied deep RL to the TORCS driving game. A 3 layer CNN is trained with an A3C strategy to learn jointly lateral and longitudinal control from a virtual front camera. The algorithm searches itself for the best driving strategy based on a reward computed from the car's angle and speed.
@cite_13 @cite_7
2260756217 1757796397
We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input. We present the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. The model is a convolutional neural network, trained with a variant of Q-learning, whose input is raw pixels and whose output is a value function estimating future rewards. We apply our method to seven Atari 2600 games from the Arcade Learning Environment, with no adjustment of the architecture or learning algorithm. We find that it outperforms all previous approaches on six of the games and surpasses a human expert on three of them.