One method to improve sample efficiency of RL by leveraging oracle policies is to initialize training with policies obtained from imitation learning, as was famously done with AlphaGo using human experts as oracle policies [1]}.
This can work reasonably well for policy gradient methods [2]}, [3]}, but can yield bad results when applied to actor-critic methods due to the critic not being well initialized on the policy [4]}, [5]}.
A similar line of work uses offline RL to initialize a learning policy by first training on prior data gathered by an oracle policy before interacting with the environment [6]}, [7]}, [8]}, [9]}.
Another related method is to carry out environment rollouts by taking composite actions that are a weighted sum of the oracle's actions and the learning policy's actions [10]}.
As training progresses, the weighting is annealed to favour the learning policy's actions over the oracle's.
