[1]} first introduces the Markov game model and proposes an algorithm analogous to value iteration for solving two-player zero-sum Markov games (with all parameters known).
Later, [2]} propose a policy iteration algorithm, and [3]} propose another policy iteration variant that works better in practice but cannot always converge. With the efforts of [4]} and [5]}, a slight variant of the [3]} algorithm is proposed in [5]} and proven to converge.
In such a full-information setting where all parameters are know, our algorithm has no estimation error and can also be viewed as a new policy-iteration algorithm.
