Most of the previous uses of epistemic uncertainty in RL have revolved around using it to motivate exploration in more uncertain states [1]}, [2]}, [3]}.
It has also been used to build risk aware RL agents, which seek to maximize expected reward without venturing into an unfamiliar part of the environment during deployment [4]}, [5]}, [6]}.
These methods utilize the variance in predicted \(Q\) -values for single \(\lbrace \mathbf {s}_t, \mathbf {a}_t\rbrace \)  pairs as a proxy to epistemic uncertainty, either via ensemble modelling or using distributional RL models.
Additionally, [3]} perform epistemic uncertainty prediction on a limited case of exploration in RL by relating it to the single-step Bellman error.
