Autonomous Platoon Control with Integrated Deep Reinforcement Learning and Dynamic Programming
Deep Reinforcement Learning (DRL) is regarded as a potential method for
car-following control and has been mostly studied to support a single following
vehicle. However, it is more challenging to learn a stable and efficient
car-following policy when there are multiple following vehicles in a platoon,
especially with unpredictable leading vehicle behavior. In this context, we
adopt an integrated DRL and Dynamic Programming (DP) approach to learn
autonomous platoon control policies, which embeds the Deep Deterministic Policy
Gradient (DDPG) algorithm into a finite-horizon value iteration framework.
Although the DP framework can improve the stability and performance of DDPG, it
has the limitations of lower sampling and training efficiency. In this paper,
we propose an algorithm, namely Finite-Horizon-DDPG with Sweeping through
reduced state space using Stationary approximation (FH-DDPG-SS), which uses
three key ideas to overcome the above limitations, i.e., transferring network
weights backward in time, stationary policy approximation for earlier time
steps, and sweeping through reduced state space. In order to verify the
effectiveness of FH-DDPG-SS, simulation using real driving data is performed,
where the performance of FH-DDPG-SS is compared with those of the benchmark
algorithms. Finally, platoon safety and string stability for FH-DDPG-SS are
demonstrated.