We consider Reinforcement Learning [1]} for solving control problems modeled as finite-horizon Markov Decision Processes (MDPs) \(\mathcal {M} = \langle \mathcal {S}, \mathcal {A}, \mathcal {R}, \mathcal {P}, \gamma \rangle \) , where \(\mathcal {S}\)  is the state space, \(\mathcal {A}\)  is the action space, \(\mathcal {R}: \mathcal {S} \times \mathcal {A} \times \mathcal {S} \rightarrow \mathbb {R}\)  is the reward function, \(\mathcal {P}: \mathcal {S} \times \mathcal {A} \rightarrow \mathcal {S}\)  is the transition kernel, and \(\gamma \in [0, 1)\)  is the discount factor. We define a policy \(\pi \in \Pi : \mathcal {S} \times \mathcal {A} \rightarrow \mathbb {R}\)  as the probability distribution of the event of executing an action \(a\)  in a state \(s\) . A policy \(\pi \)  induces a value function (VF) corresponding to the expected cumulative discounted reward collected by the agent when executing action \(a\)  in state \(s\) , and following the policy \(\pi \)  thereafter: \(Q^\pi (s,a) \triangleq \mathbb {E} \left[\sum _{k=0}^\infty \gamma ^k r_{i+k+1} | s_i = s, a_i = a, \pi \right]\) , where \(r_{i+1}\)  is the reward obtained after the \(i\) -th transition. Solving an MDP consists of finding the optimal policy \(\pi ^*\) , i.e. the one maximizing the expected cumulative discounted reward.
