Markov Decision-Process (MDP) [1]}. A canonical Data-Generating Process (DGP) assumed for OPE in reinforcement learning settings is the Markov Decision Process (MDP). MDP (finite or infinite horizon) is defined by the tuple \((\Omega _S, \Omega _A, \mathcal {P}, \mathcal {R}, p_0, \gamma )\)  where \(\Omega _S\)  indicates the state-space, \(\Omega _A\)  indicates the action-space, \(P\)  denotes the transition dynamics, i.e. \(\mathcal {P}: \Omega _S \times \Omega _A \times \Omega _S {\,\rightarrow \,}[0,1]\) . The reward is denoted here by \(Y \in \Omega _Y\) . The reward function governing the MDP is denoted by \(\mathcal {R}: \Omega _S \times \Omega _A {\,\rightarrow \,}\Omega _Y\) , \(p_0\)  the initial state distribution and the discount factor \(\gamma \) . A policy provides a distribution over action given state i.e. functionally, \(\pi : \mathcal {S}\times \mathcal {A}{\,\rightarrow \,}[0,1]\) . Let \(\mathcal {H}_{T}^{\pi } \triangleq \lbrace S_0, A_1, Y_1, \cdots , S_{t-1}, A_t, Y_t, \cdots , S_{T-1}, A_T, Y_T \rbrace \)  be the trajectories collected under policy \(\pi \)  and \(\mathcal {H}_{T}^{\pi }\)  is the collection of trajectories up to time \(T\) . Let \(G(\mathcal {H}_T^{\pi }) = \sum _{t=0}^T \gamma ^t Y_{t}\) . A behavior policy is denoted with the subscript \(\pi _b\)  and the evaluation policy is denoted by \(\pi _e\) . Figure REF (a) denotes the basic graphical model describing an MDP.
