Assumption REF  is standard in studying off-policy TD-learning algorithms [1]}, [2]}. Since we work with finite state and action spaces, under Assumption REF , the Markov chain \(\lbrace S_k\rbrace \)  admits a unique stationary distribution, denoted by \(\mu _b\in \Delta ^{|\mathcal {S}|}\)  [3]}. In addition, we have \(\Vert P^k(s,\cdot )-\mu _b(\cdot )\Vert _{\text{TV}}\le C\sigma ^k\)  for any \(k\ge 0\) , where \(C>0\) , \(\sigma \in (0,1)\)  are constants, and \(\Vert \cdot \Vert _{\text{TV}}\)  stands for the total variation distance between probability distributions [3]}. Note that in this case the random process \(\lbrace (S_k,A_k)\rbrace \)  is also a Markov chain with a unique stationary distribution, which we have denoted by \(\kappa _b\in \Delta ^{|\mathcal {S}||\mathcal {A}|}\) , and \(\kappa _b(s,a)=\mu _b(s)\pi _b(a|s)\)  for all \((s,a)\) .
