By Theorem 7.1.9 of [1]}, assum: proper implies that there is a stationary and deterministic optimal policy \(\pi ^\star \in \Pi ^{\text{\upshape {SD}}}\)  such that \(V^{\pi ^\star }(s)\ge V^{\pi }(s)\)  for any \(\pi \in \Pi ^{\text{\upshape {HD}}}\)  and any \(s\) . We let \(V^\star (\cdot )\triangleq V^{\pi ^\star }(\cdot )\)  and define
\(V_\star \triangleq \left| V^\star (s_\text{init})\right|, \quad B_\star \triangleq \max _s \left|V^\star (s)\right|.\) 
