

As shown in [1]}, \(r\)  is piecewise affine with selection functions \(r_{\pi }(V) = V - T^{\pi }V = (I-\gamma P^{\pi })V - g^{\pi }\)  for all \(\pi \in \Pi \) . Piecewise affine functions are globally Lipschitz continuous and their Lipschitz constant is given by the maximum over the norms of the coefficient matrices of their selection functions [2]}.
Therefore, by exploiting the fact that for any \(\pi \in \Pi \)  the matrices \(P^{\pi }\)  are row-stochastic, we obtain
\(&\max _{\pi \in \Pi }\Vert I - \gamma P^{\pi } \Vert _{\infty } \\&=\max _{\pi \in \Pi } \max _{s\in \mathcal {S}} \!\!\sum _{\begin{array}{c}s^{\prime }\in \mathcal {S}\setminus \left\lbrace  s\right\rbrace \end{array}}\!\!\!\! \vert -\gamma p_{ss^{\prime }}(\pi (s))\vert + \vert 1 - \gamma p_{ss}(\pi (s))\vert \\&= \max _{\pi \in \Pi } \max _{s\in \mathcal {S}} \,\gamma \left(1 - p_{ss}(\pi (s)) \right) + 1 - \gamma p_{ss}(\pi (s))\\&\le 1+\gamma \,,\) 
