The two main approaches for learning an optimal policy in an RL problem are value space methods, such as \(Q\) -learning, and policy space methods, such as AC. The \(Q\) -learning algorithm proposed in [1]} is perhaps the most well-known value space method. The asymptotic convergence of \(Q\) -learning was established in [2]}, [3]}, [4]}, [5]}. As for finite-sample bounds, see [6]}, [7]}, [8]}, [9]}, [10]} and the references therein. We next focus on related literature on AC-type of algorithms.
