Consider modelling the RL problem as an infinite horizon MDP, which consists of a finite set of states \(\mathcal {S}\) , a finite set of actions \(\mathcal {A}\) , a set of unknown transition probability matrices \(\mathcal {P}=\lbrace P_a\in \mathbb {R}^{|\mathcal {S}|\times |\mathcal {S}|}\mid a\in \mathcal {A}\rbrace \) , an unknown reward function \(\mathcal {R}:\mathcal {S}\times \mathcal {A}\mapsto \mathbb {R}\) , and a discount factor \(\gamma \in (0,1)\) . Without loss of generality we assume that \(\max _{s,a}|\mathcal {R}(s,a)|\le 1\) . For a given policy \(\pi \) , its state value function is defined by \(V^\pi (s) =\mathbb {E}_\pi [\sum _{k=0}^\infty \gamma ^k\mathcal {R}(S_k,A_k)\mid S_0=s]\)  for all \(s\in \mathcal {S}\) , and its state-action value function is defined by \(Q^\pi (s,a)=\mathbb {E}_\pi [\sum _{k=0}^\infty \gamma ^k\mathcal {R}(S_k,A_k)\mid S_0=s,A_0=a]\)  for all \((s,a)\in \mathcal {S}\times \mathcal {A}\) . The goal of RL is to find an optimal policy \(\pi ^*\)  which maximizes \(V^\pi (\mu )=\sum _s \mu (s)V^{\pi }(s)\) , where \(\mu \)  is an arbitrary fixed initial distribution over the state space. It was shown in the literature that the optimal policy is in fact independent of the initial distribution. See [1]}, [2]}, [3]} for more details for the MDP model of the RL problem.
