In RL, agents learn to solve problems by tentation; they maximize the expected rewards resulting from their actions in an environment [1]}. Environments can have complex dynamics that result in sophisticated policies that are opaque to their developers, who have no influence over what these agents learn [2]}, [3]}. Intrinsic motivation enables agents to learn behaviors that are detached from the expected rewards of the environment [4]}. It is a strategy that was developed to address the challenge of exploration in environments with sparse rewards [5]}. One such approach is Kullback-Leibler (KL) policy regularization in which the objective function is regularized by the KL-divergence between the current policy and a predefined prior [6]}. Policy regularization has been shown to be helpful and never detrimental to convergence [7]}. Although most policy regularization methods aim to improve learning performance, they can also control the learning process and imbue the policy with an intrinsic behavior [8]}. Here, the objective function is regularized with a predefined prior action distribution that defines a desirable characteristic:
\( J(\theta ) &= \mathbb {E}_{s,a \sim \mathcal {D}} \left[ R(s,a) \right] - \lambda L \\ \nonumber L &= \frac{1}{M} \sum _{j=0}^{M} \Big [ \mathbb {E}_{a \sim \pi _\theta } \left[ a_j \right] - (a_{j} \vert \pi _0(a)) \Big ]^2\) 
