For MDPs and for IMDPs with discrete action-spaces, it is well-known that policies and adversaries that solve (REF ) are Markovian (see [1]} and [2]}). Moreover, it is known that (REF ) can be solved by an iterative scheme called value iteration. The following proposition, whose proof is a straightforward extension of [3]} to the case where \(\mathcal {A}\)  is a compact set, guarantees that these results extend to caIMDPs as well:
