In the existing literature, where on-policy NAC was studied, it is typically required that all the policies achieved in the iterations of the NAC induce ergodic Markov chains over the state-action space [1]}, [2]}. Such a requirement is strong and not possible to satisfy in an MDP where the optimal policy is a unique deterministic policy. Off-policy sampling enables us to relax such an unrealistic requirement while also ensuring exploration.
