
\(J(\theta )\)  is the learning objective as a function of the model parameters \(\theta \) , \(R(s,a)\)  is the expected reward for state \(s\)  and action \(a\)  as sampled from a replay buffer \(\mathcal {D}\) , and \(\lambda \)  is a scaling hyperparameter for the regularization term \(L\) , which is the mean square difference between the observed action distribution and the action distribution given a regularization prior \(\pi _0\) . The efficacy of this approach was demonstrated by instilling an inherent characteristic behavior in agents that navigate a grid. These agents learned to either prefer left turns, right turns, or to avoid going straight by taking a zig-zag approach to their destination. In contrast to constrained RL, which avoids certain states, the policy regularization in [1]} encourages certain actions irrespective of the state and is a new direction for RL.
