PPO is a model-free, actor-critic, and policy-gradient approach to maintain data efficiency and reliable performance [1]}. In PPO, \(\pi \)  denotes the policy network optimised with its parameterisation \(\theta \) , and the policy network takes the state \(s\)  as the input and outputs an action \(a\) . PPO uses actor-critic architecture to enable learning of better policies by reformulating reward signals in terms of advantage \(A\) . The advantage function measures how good an action is compared to the other actions available in the state \(s\) . PPO maximises the surrogate objective function as follows:
\(\scalebox {1}{L(\theta )=\hat{\mathbb {E}}_t[min({r_t(\theta )\hat{A}_t,clip(r_t(\theta ),1-\epsilon ,1+\epsilon )\hat{A}_t})]}\) 
