Since the state space, action space, and horizon are all finite, there always exists an optimal policy \(\pi ^*\)  which gives the optimal value \(V_h^*(x) := \text{sup}_\pi V_h^\pi (x)\) , for all \(x \in \mathcal {S}\)  and \(h \in [H]\)  [1]}.
To simplify notation we will denote \([\mathbb {P}_h V_{h+1}](x, a) := \mathbb {E}_{x^{\prime }\sim \mathbb {P}(\cdot |x, a)} V_{h+1}(x^{\prime })\) . Using this notation we have the Bellman equations and Bellman optimality equations as follows:
\({\left\lbrace \begin{array}{ll}V_h^\pi (x) = Q_h^\pi (x, \pi _h(x))\\Q_h^\pi (x, a) = (r_h + \mathbb {P}_h V_{h+1}^{\pi })(x, a)\\V_{H+1}^\pi (x) = 0 \quad \forall x \in \mathcal {S}\end{array}\right.} \\{\left\lbrace \begin{array}{ll}V_h^*(x) = \text{max}_{a \in \mathcal {A}} Q_h^*(x, a)\\Q_h^*(x, a) := (r_h + \mathbb {P}_h V_{h+1}^*)(x, a)\\V_{H+1}^*(x) = 0 \quad \forall x \in \mathcal {S}\end{array}\right.}\) 
