To proceed and establish finite-sample bound of Algorithm (REF ), we will apply Markovian SA results in the literature. In particular, we will apply Theorem 2.1 of [1]}, which is presented in Appendix REF  for self-containedness. To achieve that, we establish properties of the operators \(F(\cdot )\) , \(\bar{F}(\cdot )\) , and the Markov chain \(\lbrace Y_k\rbrace \)  in the following proposition, which guarantee that all the assumptions for applying [1]} is satisfied. The proof is presented in Appendix REF .
