Here \(\hat{\operatornamewithlimits{\mathbb {E}}}_h[f(s_h,a_h)]\)  indicates the sample average approximation to \(\operatornamewithlimits{\mathbb {E}}_{\begin{array}{c}s\sim \rho ^{\pi _\theta }\\a\sim \pi _\theta (\cdot |s)\end{array}} \left[f(s,a)\right]\)  where \(\rho ^{\pi _\theta }\)  is the distribution of states induced by the policy. The terms \(L_h^{\text{clip}}(\theta )\) , \(L_h^{VF}(\theta )\)  and \(S[\pi _\theta ](s_h)\)  are the standard PPO-clip, value function and entropy exploration bonus terms, respectively, taken directly from [1]}. Only the first two terms are required for PPO, we include the entropy term as it has been shown to improve exploration by discouraging premature convergence to suboptimal policies [2]}, [3]}. The last term in the objective is a new contribution in this work for bounding the learned action distribution, given by
\(\mu _\theta ^+(s_h) = \hat{\operatornamewithlimits{\mathbb {E}}}_{h}\left[\sum _{j=1}^{\dim (a)} \max \left(\left|\mu _{j,\theta }(s_h)\right|-(1-\epsilon ), 0\right)^2 \right]\) 
