Consider a set of \(N\)  agents acting concurrently in the same environment and transmitting their state-action information to a learner. For each agent, the setup is modeled as a 5-tuple \((S,A,P,R,\gamma )\) . Here \(S\)  is the finite state space, \(A\)  is the finite action space, \(P\)  is the dynamics probability law (given state \(s\)  and action \(a\) , the probability to traverse to state \(s^{\prime }\)  is \(p(s^{\prime }|s,a)\) ), \(R(s,a,s^{\prime })\)  is the reward function, and \(\gamma \)  is the discount factorThe results in this paper hold, with appropriate modifications, for a a continuous state space, but we focus for simplicity on the finite state space.. This is a specific case of a dynamic Stochastic Game [1]} in which agents are cooperative (having the same reward function) and have the same set of actions. The \(Q\) -function under policy \(\pi \)  for some agent is defined as \(Q^{\pi }(s,a)=E^{\pi }\left[\sum _{t=0}^{\infty } \gamma ^{t}R(s_{t},a_{t},s_{t+1})|s_{0}=s,a_{0}=a\right]\) , and the optimal \(Q\) -function is \(Q^{*}(s,a)=\max _{\pi }Q^{\pi }(s,a)\) . The Bellman operator for \(\pi \)  is \(B^{\pi }Q(s,a)=\sum _{s^{\prime }} p(s^{\prime }|s,a)[R(s,a,s^{\prime })+\gamma Q(s^{\prime },\pi (s^{\prime }))]\) . Similarly, the value function for \(\pi \)  is defined as \(V^{\pi }(s)=E^{\pi }[\sum _{t=0}^{\infty } \gamma ^{t}R(s_{t},a_{t},s_{t+1})|s_{0}=s]\) .
We also assume that all \(Q\) -functions are bounded in the interval \([0,Q_{\max }]\) , where \(Q_{\max }=R_{\max }/(1-\gamma )\) .
<FIGURE>