In this setting, there are many centralized algorithms developed in a line of recent works with near-optimal sample complexity for finding a Nash equilibrium [1]}, [2]}, [3]}, [4]}, [5]}, [6]}.
These algorithms require a central controller that collects some global knowledge (such as the actions and the rewards of all players) and then jointly decides the policies for all players.
Centralized algorithms are usually convergent (as defined in [7]}), in the sense that the policies of the players converge to the set of Nash equilibria.
