Reinforcement learning (RL) methods achieved major advances in multiple tasks surpassing human performance. However, most of RL strategies show a certain degree of weakness and may become computationally intractable when dealing with high-dimensional and non-stationary environments. In this paper, we build a meta-reinforcement learning (MRL) method embedding an adaptive neural network (NN) controller for efficient policy iteration in changing task conditions. Our main goal is to extend RL application to the challenging task of urban autonomous driving in CARLA simulator. "Every living organism interacts with its environment and uses those interactions to improve its own actions in order to survive and increase" BID13 . Inspired from animal behaviorist psychology, reinforcement learning (RL) is widely used in artificial intelligence research and refers to goal-oriented optimization driven by an impact response or signal BID30 . Properly formalized and converted into practical approaches BID9 , RL algorithms have recently achieved major progress in many fields as games BID18 BID28 and advanced robotic manipulations BID12 BID17 beating human performance. However, and despite several years of research and evolution, most of RL strategies show a certain degree of weakness and may become computationally intractable when dealing with high-dimensional and non-stationary environments BID34 . More specifically, the industrial application of autonomous driving in which we are interested in this work, remains a highly challenging "unsolved problem" more than one decade after the promising 2007 DARPA Urban Challenge BID2 ). The origin of its complexity lies in the large variability inherent to driving task arising from the uncertainty of human behavior, diversity of driving styles and complexity of scene perception.An interpretation of the observed vulnerability due to learning environment changes has been provided in contextaware (dependence) research assuming that "concepts in the real world are not eternally fixed entities or structures, but can have a different appearance or definition or meaning in different contexts" BID36 . There are several tasks that require context-aware adaptation like weather forecast with season or geography, speech recognition with speaker origins and control processes of industrial installations with climate conditions. One solution to cope with this variability is to imitate the behavior of human who are more comfortable with learning from little experience and adapting to unexpected perturbations. These natural differences compared to machine learning and specifically RL methods are shaping the current research intending to eschew the problem of data inefficiency and improve artificial agents generalization capabilities BID10 . Tackling this issue as a multi-task learning problem BID3 , meta-learning has shown promising results and stands as one of the preferred frames to design fast adapting strategies BID25 BID23 . It refers to learn-to-learn approaches that aim at training a model on a set of different but linked tasks and subsequently generalize to new cases using few additional examples BID7 .In this paper we aim at extending RL application to the challenging task of urban autonomous driving in CARLA simulator. We build a meta-reinforcement learning (MRL) method where agent policies behave efficiently and flexibly in changing task conditions. We consolidate the approach robustness by integrating a neural network (NN) controller that performs a continuous iteration of policy evaluation and improvement. The latter allows reducing the variance of the policy-based RL and accelerating its convergence. Before embarking with a theoretical modeling of the proposed approach in section 3, we introduce in the next section metalearning background and related work in order to better understand the current issues accompanying its application to RL settings. In the last section, we evaluate our method using CARLA simulator and discuss experimental results. In this paper we addressed the limits of RL algorithms in solving high-dimensional and complex tasks. Built on gradient-based meta-learning, the proposed approach implements a continuous process of policy assessment and improvement using a NN controller. Evaluated on the challenging problem of autonomous driving using CARLA simulator, our approach showed higher performance and faster learning capabilities than conventionally pre-trained and randomly initialized RL algorithms. Considering this paper as a preliminary attempt to scale up RL approaches to high-dimensional real world applications like autonomous driving, we plan in future work to bring deeper focus on several sides of the approach such as the reward function, CNN architecture and including vehicle characteristics in the tasks complexity setup.
A meta-reinforcement learning approach embedding a neural network controller applied to autonomous driving with Carla simulator.
{"src": "scitldr-AIC_train", "paper_id": "S1eoN9rsnN"}