To explore an environment during optimization, most popular methods generally employ stochastic actions to explore the environment [1]}, [2]}, [3]}, [4]}, [5]}, [6]}, [7]}.
As a result, at the beginning of training, a randomly initialized policy takes random actions as it has no understanding of the environment.
In most environments, it is usually easy to create a simple policy that performs better than random via methods other than RL, e.g. hard-coding a policy to produce very basic behavior. We refer to this as an oracle policy.
If an RL agent was able to learn from an oracle policy instead of trying random actions at the beginning of training, the number of samples required for its training could be reduced and/or its learning could be enhanced by improving the agent's ability to escape local minima.
This idea of bootstrapping from an oracle is hereafter referred to as Critic Confidence Guided Exploration (CCGE).
In order to incorporate CCGE into an actor-critic algorithm, we theoretically derive and implement a method to estimate the uncertainty of the critic's predictions and use that measure to control its influence during training. In other words, the oracle is favoured when uncertainty is high, with the critic gaining influence as the learning progresses.
