Proximal policy optimization, PPO [1]} is an improvement to Trust region policy optimization, TRPO [2]}, a policy-gradient-based method. TRPO could improve the solution to many of the complex continuous learning problems. TRPO's idea emerged from the fact that previously, to achieve the best solutions, many of these kinds of tasks had been solved by precisely adjusting the reward function. However, this approach led to results that would respond poorly to the slightest change to the problem's setting. Thus, the intent was to make the agent capable of acting with the same optimality while receiving a simple reward signal. The approach taken is to confront the agent with a spectrum of challenges in the environment with a growing difficulty. To make the idea beneficial, the information from recent experiences must not disregard early ones. In TRPO, to make the policy gradient robust, updated policy's change from the old one, in terms of Kullback–Leibler divergence, is bound in a trust region [2]}. PPO [1]}, as an approximation of TRPO, restricts the policy gradient via a KL-regularization term. The regularization term has a coefficient that increases by a scale factor if the policy update goes beyond the upper bound and decreases in the same way if it falls below the lower bound. In contrast with TRPO, PPO eases the use of the framework in an extensively parallelized framework and with RNNs, due to the sole reliance on first-order gradients.
\(J_{\textrm {PPO}}(\phi ) = J(\phi ) - \lambda \textrm {KL}[\pi _{\textrm {old}}|\pi _\phi ]\) 
