In this paper, we develop a RL-based controller which is trained by the the proximal policy optimization (PPO) for a traffic system compose of \(2\times 2\)  PDEs with input delay and show the controller is robust to parameter disturbance.
Proximal policy optimization (PPO) [1]} is chosen over other policy gradient methods, such as trust policy optimization (TRPO) [1]} or deep deterministic policy gradient (DDPG) algorithm [3]}, because PPO is more robust to the step size and shows higher performance and lower computational complexity.
Unlike the boundary ramp metering control in [4]}, where the PPO trains boundary inputs, i.e., one value per step, in this study, the time-gap of the ACC-equipped vehicle is used as manipulation values which are in-domain inputs. It is difficult for the PPO to train so many inputs of all the in-domain points. Inspired by the Beckstepping controller but much simpler than that of the Backstepping segmented controller, we take a weighted sum of three feedbacks, the feedback from the current two states and the feedback from the previous step input. The gains of the controller are trained through the interaction between the PPO and the numerical traffic system simulator.
