PPO is a model-free, policy-gradient method, whose hyperparameters are robust to multiple tasks. Also, PPO algorithm exhibits high performance and low computational complexity [1]} compared to other policy gradient methods. The explicit knowledge of the traffic model is not required for PPO-based control approaches. In the paper, we simulate traffic dynamics by discretizing the nonlinear model by the finite difference method and use this numerical simulator as the environment with which the PPO interacts. The solution \(\rho (x,t)\)  and \(v(x,t)\)  to (REF )-() is discretized in domain \([0,L]\times [0,T]\) . The discretization space step is denoted by \(\Delta x=L/M\)  and time step is denoted by \(\Delta t=T/N\) , which are chosen such that the Courant-Friedrichs-Lewy condition is met, namely \(\frac{\Delta x}{\Delta t} \ge \max |\lambda _{1,2}|\) , where \(\lambda _{1,2}\)  are the characteristic speed of the model (REF )-(). Hence, the discretized states and input at time \(t\)  can be written as:
\(\rho ^{t}=&[\rho (x_0,t),\rho (x_1, t),...,\rho (x_m,t)],\\v^{t}=&[v(x_0,t),v(x_1, t),...,v(x_m,t)],\\h_{\rm {acc}}^{t-1}=&[h_{\rm {acc}}(x_0,t-1),h_{\rm {acc}}(x_1,t-1),...,\nonumber \\&~h_{\rm {acc}}(x_m,t-1)],\) 
