Trust Region Policy Optimization (TRPO) [1]} relaxes this restrictive linear mixing
by using the Total Variation (TV) divergence between two stochastic policies for regularization.
The same regularization was also adopted in Constrained Policy Optimization (CPO) [2]}.
With the TV divergence as regularization,
TRPO and CPO present two types of monotonic improvement guarantee,
one relying on the maximum TV over all state-action pairs [1]} and
the other relying on the average TV divergence over all empirical samples [2]}.
As TV divergence can be upper bounded by KL divergence,
in practice, TRPO relies on the KL divergence to update the policies.
This KL divergence has thus been used for theoretical analysis in many studies,
e.g., [5]}, [6]}.
