Kullback-Leibler (KL) regularized RL integrates an oracle policy by incorporating the oracle's actions into the actor's action distribution using an additional KL loss between the two distributions.
This has been shown to work on actor critic methods on a range of RL benchmarks [1]}, [2]}, [3]}, [4]}, [5]}, [6]}.
However, in some cases, this can cause the actor to perform poorly when trying to assign probability mass to the singular action given by the oracle over a distribution of actions [7]}.
