
Here \(\mathcal {T}_\pi ^n(\cdot )\)  denotes the \(n\) -step Bellman operator, and \(\Pi _{\kappa _b}(\cdot )\)  stands for the projection operator onto the linear sub-space \(\mathcal {Q}\)  with respect to the weighted \(\ell _2\) -norm with weights \(\lbrace \kappa _b(s,a)\rbrace _{(s,a)\in \mathcal {S}\times \mathcal {A}}\)  [1]}. It is well-known that the operator \(\mathcal {T}_\pi (\cdot )\)  (i.e., the one-step Bellman operator) is a contraction mappingIt is also known that \(\mathcal {T}_\pi (\cdot )\)  is a contraction mapping with respect to the weighted \(\ell _2-\)  norm \(\Vert \cdot \Vert _\kappa \) , where \(\kappa \in \Delta ^{|\mathcal {S}||\mathcal {A}|}\)  is the stationary distribution of the Markov chain \(\lbrace (S_k,A_k)\rbrace \)  under the target policy \(\pi \)  [1]}. However, since we do not assume that the target policy induces an ergodic Markov chain, such \(\kappa \)  may not be unique and/or may not induce a norm. Hence we cannot use this contraction property here. with respect to \(\Vert \cdot \Vert _\infty \) , with contraction factor \(\gamma \) . Moreover, the projection operator \(\Pi _{\kappa _b}(\cdot )\)  is a non-expansive operator with respect to the weighted \(\ell _2\) -norm \(\Vert \cdot \Vert _{\kappa _b}\) . However, due to the norm mismatch, the composed operator \(\Pi _{\kappa _b}\mathcal {T}_\pi (\cdot )\)  need not be a contraction mapping with respect to either \(\Vert \cdot \Vert _\infty \)  or \(\Vert \cdot \Vert _{\kappa _b}\) . Specifically, for any given \(Q_1\)  and \(Q_2\) , in general we only have
\(\Vert \Pi _{\kappa _b}\mathcal {T}_\pi (Q_1)-\Pi _{\kappa _b}\mathcal {T}_\pi (Q_2)\Vert _{c}\le (\gamma /\sqrt{\kappa _{b,\min }}) \Vert Q_1-Q_2\Vert _{c},\) 
