
Based on the above approximation steps, the optimization of the original policy in (REF ) is transformed into a conjugate gradient form which allows for estimating the expectations of the policy objective in (REF ) with policy improvement guarantees. In this regard, the update direction can be approximated by \(\approx \hat{H}^{-1}\nabla _\theta J(\pi _\theta ),\)  where \(\hat{H}^{-1}\)  is the Hessian-vector product of sampled KL-divergence. Finally, the actor parameter is updated via backtracking line search [1]} subject to the KL constraint as follows:
\(\theta _{t+1} = \theta _t + _t^a \sqrt{\frac{2\epsilon _{KL}}{^\top \hat{H} }} ,\) 
