Advance \(N\)  agents
[1]
Initialize: Adjacency matrix \(E=\left[e_{ij}\right]\)  and Weight matrix \(W=\left[w_{ij}\right]\) 
\(i\leftarrow 1\)  to \(N\) , from state supdate is true \(\forall (s,a)\)  Linear estimation of the \(Q\) -function
Receive \(\widetilde{Q}_{L}^{i}(s,a)\)  from the learner, and \(d_{i}\)  samples \(\lbrace \widetilde{Q}_{A}^{j,i}(s,a)\rbrace _{j \ne i}\)  from agents.
set \(\hat{Q}^{i}(s,a) \leftarrow w_{ii}\widetilde{Q}_{L}^{i}(s,a) + \sum _{j \ne i}^{N}e_{ji}w_{ji}\widetilde{Q}_{A}^{j,i}(s,a)\) 
set \(\hat{Q}^{i}_{sat}(s,a) \leftarrow \max \lbrace Q_{max},\min \lbrace 0,\hat{Q}^{i}(s,a)\rbrace \rbrace \) 
set update to false.
set update to false.
Perform action \(a=\operatornamewithlimits{argmax}_{\widetilde{a}}\hat{Q}_{sat}^{i}(s,\widetilde{a})\) .
Receive reward \(R\)  and transition to state \(s^{\prime }\) .
Transmit a copy of \((s,a,\widetilde{R},s^{\prime })\)  to the a list \(Z\)  of the learner. Reward is noisy


The main contribution of this section is to establish PAC guarantees for the noisy communication setting, which we believe to be the first result for this case. Space and computational complexity guarantees and detailed proofs can be found in the Appendix.
Before presenting the main sample complexity theorem, we describe the noise setup. In this work we focus on two sources of noise - additive noise and quantization noise. A white Gaussian additive noise is usually used to describe the channel noise resulting from thermal fluctuations in the receiver's circuits, and is the main limitation in space and satellite communication [1]}, [2]}. Quantization noise models analog to digital quantization performed at the receiver, and in some cases can be modeled as an orthogonal noise term which is bounded by the interval between quantization levels
[2]}, [4]}. Although there are many other sources of communication limitations in distributed systems, such as latency [5]} and communication failures [6]}, in this work we focus on additive noise and quantization noise in order to stress resulting theoretical phenomena resulting from communication constraints, and do not necessarily view our algorithm as intended for some specific application. Each agent \(i\)  receives a noisy version of the \(Q\) -function as in (REF ) every time the learner updates its real \(Q\) -function, such that the communication noise \(n^{i}(s,a)\)  is composed of an additive noise term \(b\)  and a quantization term \(m\) ,
\(\begin{aligned}n^{i}(s,a) &= b^{i}(s,a)+m^{i}(s,a)~.\end{aligned}\) 
