While several existing works have focused on developing control and verification algorithms for finite-space and finite-action IMDPs [1]}, [2]}, [3]}, efficient algorithms for control of IMDPs with continuous action-spaces are missing. A first attempt to close this gap has been proposed in [4]}, where the authors present an algorithm to synthesize policies that maximize a reachability probability based on suboptimal heuristics and non-convex optimization. In contrast, here we derive an exact reformulation of value iteration for caIMDPs, which leads to tractable solutions based on linear or convex programming, in many cases of interest. caIMDPs are also closely related to parametric MDPs, which are Markov decision processes with finite state and action spaces, but where the transition probabilities may depend on some parameters [5]}, [6]}, [7]}. However, a key difference is that in parametric MDPs, the problem is to find parameter values that are fixed and independent of the state of the system. Instead, in caIMDPs we seek feedback control policies that depend on time and state, thus requiring substantially different approaches.
