After obtaining the actions \((a_k^s(t),a_k^p(t))\)  based on the observed state \(s_k(t)\)  at time slot \(t\) , the immediate reward \(r_k(t)\)  and the new state \(\mathbf {s}_k(t+1)\)  will be returned to the agent \(k\)  from the environment. Afterwards, the experience \((\mathbf {s}_k^t,\mathbf {a}_k(t),r_k(t),\mathbf {s}_k(t+1))\)  obtained at time slot \(t\)  for the agent \(k\)  is stored in the replay memory block \(\mathcal {D}\)  with size \(U\)  by using the experience replay strategy [1]}. Then, a mini-batch of experiences \((\mathbf {s}_k(i),\mathbf {a}_k(i),r_k(i),\mathbf {s}_k(i+1))\)  with size \(N_{tr}\)  is randomly selected by the agent \(k\)  from the replay memory. Notice that the selected experiences may include the experiences of different agents at different time slots. The aim of using the mini-batch of experiences rather than the current time slot experience is to ensure that the data used for training is independently and identically distributed. The symbol \(i\)  in the bracket denotes the experience of the \(i\) -th time slot. Based on the states \(\mathbf {s}_k(i)\)  and \(\mathbf {s}_k(i+1)\)  selected in the mini-batch, the output \(Q\) -value at the \(Q\)  network and target \(Q\)  network can be expressed as \(Q(\mathbf {s}_k(i),a_k^s(i),\mathbf {\theta })\)  and \(\tilde{Q}(\mathbf {s}_k(i+1), a_k^s(i+1),\tilde{\mathbf {\theta }})\) , respectively, where \(\tilde{\mathbf {\theta }}\)  is the weights for the target \(Q\)  network. The difference of the output \(Q\)  value between \(Q\)  and target \(Q\)  networks can be measured by using the following loss function
\(L_k^{D}(\mathbf {\theta })&=\sum _i(r_k(i)+\max _{a_k^s(i+1)\in \mathbf {\mathcal {A}}_k}\tilde{Q}(\mathbf {s}_k(i+1),a_k^s(i+1),\tilde{\mathbf {\theta }})\nonumber \\&~~~~~~~~-Q(\mathbf {s}_k(i),a_k^s(i),\theta ))^2,\forall k.\) 
