Autonomous driving is still considered as an “unsolved problem” given its inherent important variability and that many processes associated with its development like vehicle control and scenes recognition remain open issues. Despite reinforcement learning algorithms have achieved notable results in games and some robotic manipulations, this technique has not been widely scaled up to the more challenging real world applications like autonomous driving. In this work, we propose a deep reinforcement learning (RL) algorithm embedding an actor critic architecture with multi-step returns to achieve a better robustness of the agent learning strategies when acting in complex and unstable environments. The experiment is conducted with Carla simulator offering a customizable and realistic urban driving conditions. The developed deep actor RL guided by a policy-evaluator critic distinctly surpasses the performance of a standard deep RL agent. An important approach for goal-oriented optimization is reinforcement learning (RL) inspired from behaviorist psychology BID25 . The frame of RL is an agent learning through interaction with its environment driven by an impact (reward) signal. The environment return reinforces the agent to select new actions improving learning process, hence the name of reinforcement learning BID10 . RL algorithms have achieved notable results in many domains as games BID16 and advanced robotic manipulations BID13 beating human performance. However, standard RL strategies that randomly explore and learn faced problems lose efficiency and become computationally intractable when dealing with high-dimensional and complex environments BID26 .Autonomous driving is one of the current highly challenging tasks that is still an "unsolved problem" more than one decade after the promising 2007 DARPA Urban Challenge BID4 ). The origin of its difficulty lies in the important variability inherent to the driving task (e.g. uncertainty of human behavior, diversity of driving styles, complexity of scene perception...).In this work , we propose to implement an advantage actor-critic approach with multi-step returns for autonomous driving. This type of RL has demonstrated good convergence performance and faster learning in several applications which make it among the preferred RL algorithms BID7 . Actor-critic RL consolidates the robustness of the agent learning strategy by using a temporal difference (T D) update to control returns and guide exploration. The training and evaluation of the approach are conducted with the recent CARLA simulator BID6 . Designed as a server-client system, where the server runs the simulation commands and renders the scene readings in return, CARLA is an interesting tool since physical autonomous urban driving generates major infrastructure costs and logistical difficulties. It particularly offers a realistic driving environment with challenging properties variability as weather conditions, illumination, and density of cars and pedestrians.The next sections review previous work on actor-critic RL and provide a detailed description of the proposed method. After presenting CARLA simulator and related application advantages, we evaluate our model using this environment and discuss experimental results. In this paper we addressed the limits of RL algorithms in solving high-dimensional and complex tasks. Combining both actor and critic methods advantages, the proposed approach implemented a continuous process of policy assessment and improvement using multi-step T D learning. Evaluated on the challenging problem of autonomous driving using CARLA simulator, our deep actor-critic algorithm demonstrated higher performance and faster learning capabilities than a standard deep RL. Furthermore, the results showed a certain vulnerability of the approach when facing unseen testing conditions. Considering this paper as a preliminary attempt to scale up RL approaches to highdimensional real world applications like autonomous driving, we plan in future work to examine the performance of other RL methods such as deep Q-learning and Trust Region Policy Optimization BID22 on similar complex tasks. Furthermore, we propose to tackle the issue of non-stationary environments impact on RL methods robustness as a multi-task learning problem BID5 . In such context, we will explore recently applied concepts and methodologies such as novel adaptive dynamic programming (ADP) approaches, context-aware and meta-learning strategies. The latter are currently attracting a keen research interest and particularly achieving promising advances in designing generalizable and fast adapting RL algorithms BID21 BID20 . Subsequently, we will be able to increase driving tasks complexity and operate conclusive comparisons with the few available state-of-the-art experiments on CARLA simulator.
An actor-critic reinforcement learning approach with multi-step returns applied to autonomous driving with Carla simulator.
{"src": "scitldr-AIC_train", "paper_id": "Bke03G85DN"}