[Proof of Corollary REF ]
The fact that maximizing a convex function over a polytope attains a solution at the vertices is well-known; e.g., see [1]}.

In this case, we can either solve all \(|\mathcal {Q}|\)  problems in (REF ), by evaluating the objective function on the vertices of \(\mathcal {A}\) , or we can solve the \(|\mathrm {ver}(\mathcal {A})|\)  LPs (REF ), which, again, is equivalent to building a discrete-action IMDP, as explained in Remark REF . Either way, when \(|\mathrm {ver}(\mathcal {A})|\)  is small, value iteration can be solved quickly.
