A discrete-time hybrid factored MDP (HMDP) [1]} with a finite horizon is a tuple, \(\mathcal {M} = (\mathbf {X}, A, R, P, D)\) . \(D = \lbrace t_0, t_1, \ldots , t_H \rbrace \)  is a finite sequence of decision epochs, or time steps.
For simplicity, we assume that the time step size, \(\Delta t=t_{i+1} - t_i\) , is a constant.
\(\mathbf {X}\)  is a state space represented by a set of state variables, \(\lbrace X_1, X_2, \ldots , X_n \rbrace \) .
A state is defined by a vector \(\mathbf {x}\)  of assignments to each state variable, which splits into discrete and continuous components denoted by \(\mathbf {x} = (\mathbf {x}_D, \mathbf {x}_C)\) .
For any state vector, \(\mathbf {x}\) , we write \(\mathbf {x}[j]\)  to refer to the value of state variable \(j\) .
\(A\)  is a finite set of actions.
\(R : \mathbf {X} \times A \times D \rightarrow \mathbb {R}_{\ge 0}\)  is a non-negative reward function.
\(P : \mathbf {X} \times A \times D \times \mathbf {X} \rightarrow \mathbb {R}_{\ge 0}\)  is a time-varying transition function which describes the transition dynamics conditioned upon the previous state, action, and time step.
A deterministic Markovian policy is a mapping from the state and time step to an action: \(\pi : \mathbf {X} \times D \rightarrow A\) .
The objective is to find a policy which optimises the expected total reward over the finite horizon, \(\mathbb {E}[\sum _{t=t_0}^{t_H}R(\mathbf {x}_t, \pi (\mathbf {x}_t))]\) . The optimal value function [2]}, \(V_{\pi ^*}\) , satisfies
