This paper makes five contributions.
First, we derive a quantity that we term pathwise epistemic uncertainty, which offers a formal metric of the epistemic uncertainty of the \(Q\) -value predictions during training (Section ).
Second, we derive a quantity that we term the \(F\) -value that is a learnable upperbound value to the pathwise epistemic uncertainty metric.
Third, we propose a new strategy to improve sample efficiency via the use of oracle policies, which we term CCGE (Section REF ).
Fourth, we show a way to implement this metric in deep RL for Q-learning and for actor-critic based methods (Section REF ). As expected, our metric is observed to provide insight into model learning, such as when an agent discovers a new high-reward state (Section REF ).
Fifth, we implement one form of CCGE using \(F\) -values and show that it can improve sample efficiency in PyBullet environments [1]} as well as reduce learning variance in a hyperparameter sensitive, domain-randomized Gym CarRacing environment [2]} (Section REF ).
