
(If any of the denominator is zero, define the corresponding estimator as zero.)
To make sure that these are accurate estimators for every state, we naturally need to ensure that every state is visited often enough.
To this end, we make the following assumption similar to [1]}, which essentially requires that the induced Markov chain under any stationary policy pair is irreducible.
