A Markov game [1]} is a tuple \(\mathcal {G} = (S, n, \mathcal {A}, O, T, r)\) , where \(S\)  is a set of states \(s\) , \(n\)  is a number of players, \(\mathcal {A}_i\)  is a set of actions \(a\)  of player \(i\) . \(\mathcal {O}: S \times N \rightarrow \mathbb {R}^d\)  is an observation function that specifies d-dimensional observations available to each agent and \(O_i\)  is set of observations \(o_i\)  of agent \(i\) . \(T: S \times \mathcal {A}_1 \times \dots \times \mathcal {A}_n \rightarrow \Delta (S)\)  and \(r_i: S \times \mathcal {A}_1 \times \dots \times \mathcal {A}_n \rightarrow \mathbb {R}\)  are transition and reward functions respectively, where \(\Delta \)  is a discrete probability distribution over \(S\) . We define \(\Delta _0(S)\)  as the distribution of initial states and return of player \(i\)  in state \(s\)  with discount factor \(\gamma \in [0, 1)\)  as \(R_i(s) = \overset{\infty }{\underset{t=0}{\sum }} \left[ \gamma ^t r_i(s_t, a^1_{t}, \dots , a^n_{t}) \mid s_0 = s \right]\) . Each player has its own policy \(\pi _i: O_i \rightarrow \Delta (\mathcal {A}_i)\)  that returns probability for each action to be taken when observing \(o_i\)  and value function \(V_{i}(s) = \mathop {\mathbb {E}}_{\pi _i}[R_i(s)]\) . We also define Q-function as \(Q_{i}(s,a) = \mathop {\mathbb {E}}_{\pi _i}[R_i(s) \mid a^i_{0} = a]\) . Bold font will denote vector, e.g. \(\textbf {a} = (a^1, \dots , a^N)\)  and subscript \(a_t\)  will denote current step in the environment.
