,
where \(\theta \)  is the policy parameter (the set that contains all weights and biases), \(\hat{E}_t\)  is the expectation over time steps, \(r_t\)  is the ratio of the probability under the new and old policies, \(\hat{A}_t\)  is the estimated advantage, and \(\epsilon \)  is the hyperparameter for bounding the clipping range. There is also a variant of PPO based on an adaptive Kullback-Leiber penalty, which controls the change of \(\pi (a|s)\)  at each iteration. A detailed explanation of PPO, as well as its pseudocodes, are already clearly presented in the original paper [1]}. Although there are arguments about the origin of performance enhancement from Trust Region Policy Optimization [2]} (whether it is from the clipping or code-level tricks [3]}), we reckon these topics, including if PPO-like algorithms can be further optimized, go beyond the scope of this work. Thus, we implement a minimal PPO for our quantum control task.
