Regularized policy optimization learns a policy directly with a regularized objective function.
Kakade and Langford kakade2002approximately show that,
with the objective defined as policy advantage, a.k.a. a surrogate objective,
one could improve a policy by linearly mixing it
with the policy that optimizes the surrogate objective,
i.e., Conservative Policy Iteration (CPI) [1]}.
[2]} propose the optimal linear mixing that
guarantees a robust monotonic improvement.
