Proposition 6
For a non-determinstic MDP, the upper bound of the stepwise epistemic uncertainty derived in Proposition 4 (which differs from the true value by a constant) can be estimated using Monte Carlo sampling.
The corresponding proof is in Appendix REF , which assumes that the \(Q\) -value is Gaussian.
Though restrictive, this is not the first time this assumption has been made [1]}, [2]}, and a case in distributional RL has also made this assumption [3]}.
\( \begin{aligned}\hspace{20.0pt}&\hspace{-20.0pt}\mathcal {E}_t(Q^\pi _\phi , \lbrace \mathbf {s}_t, \mathbf {a}_t\rbrace ) \\&= \mathbb {E}[l(Q^\pi _\phi (\mathbf {s}_t, \mathbf {a}_t) - r_t - \gamma V^\pi _\phi (\mathbf {s}_{t+1}))]+ \sigma ^2(r_t) \\&\ge \mathbb {E}[l(Q^\pi _\phi (\mathbf {s}_t, \mathbf {a}_t) - r_t - \gamma V^\pi _\phi (\mathbf {s}_{t+1}))]\end{aligned}\) 
