In this equation, \(Q(s,r)\)  is approximated to \(Q(s,r,\theta )\) , where \(\theta \)  presents the weights of deep neural network used for the DQN training.
To stabilize this training and enable the system to converge to the optimal learning, different steps should be followed, which are detailed in algorithm REF . The algorithm is initiated by creating two identical Q-Networks using the same NN weights (lines 2-3). Then, during the learning process, the system generates repeatedly the episodes of experiences (\(S_t\) ,\(A_t\) ,\(R_t\) ,\(S_{t+1}\) ) (lines 5-18) that are stored constantly in a buffer \(D\)  (lines 19-21). The decision taking process follows an \(\epsilon -\)  greedy algorithm (line 10). This algorithm allows the agent to take actions either randomly with a probability \(\epsilon \)  or by selecting the highest Q-value (\(A_t=\) arg max\(Q(S_t,A_t)\) ) experienced until the current time step. More specifically, the RL agent starts by exploring the environment with a probability \(\epsilon \)  equal to 1, and then this probability decays over time to enable the system to use the past learned knowledge. At the end of the training process, although \(\epsilon \)  becomes small, it should not be equal to 0 to allow the system to take some random actions in order to discover environment changes and adapt to it. To improve the Q-values, the agent randomly samples at each time step a minibatch from the replay buffer D (line 22), and calculates the target values for each sample in the batch using the Q-network (line 24). This mechanism, called experience replay, contributes to reach the convergence of the system by learning from past experience. Moreover, to stabilize the learning, the agent keeps the target network fixed, while improving the main Q-network. Then, after each \(G\)  steps, this target network is updated towards the most recent learning presented by the main network (line 27) [1]}.
[h]
RL-DistPrivacy

[1]
DQN initialization:
Initialize the Q-network and the target network with the
same random parameters \(\theta \)  and \(\theta ^{\prime }\) , where \(\theta ^{\prime } \leftarrow \theta \) .
DQN Learning:
each \(r\in RQ\) 
each episode \(l^{k}_{r^*}\in L_{r^*}\) 
each time-step \(t=1..p_{l^{k}_{r^*}}\) 
\(S_t=\lbrace r^*, l^{k}_{r^*}, p_{l^{k}_{r^*}}, \lbrace \bar{c_1},...,\bar{c_D}\rbrace , , \lbrace \bar{b_1},...,\bar{b_D}\rbrace ,\) 
\(\lbrace \bar{m_1},...,\bar{m_D}\rbrace , \lbrace \bar{L_1},...,\bar{L_D}\rbrace ,\lbrace A_1,...,A_{P_{l^{k-1}_{r^*}}}\rbrace  \rbrace \) 
Select \(A_t\)  based on \(\epsilon -\) greedy policy
\(\sum A_t=1\) 
\(j=1:D\) 
\(t>1\)  and \(t<L_{r^*}\) 
\(R_t=R_t -A_t(j)\times (\frac{O^{k-1}_{i,j}}{\rho _i}\times \mathrm {1}_{I_i\ne I_j}+\) 
\(c_{p,k}^k/e(j))+\beta _j\) 
\(C_1 \times C_2 \times C_3=1\) 
\(R_t=R_t+max(1,\sigma \times \sum \limits _{j=1}^{D}A_t(j)\) 
\(\times \sum \limits _{l=1}^{t}A_l(j))\) 
(1) Receive \(R_t\)  and observe the next state \(S_{t+1}\) .
(2) Store \((S_{t},A_t,R_t,S_{t+1})\)  in the replay
memory \(D\) .
(3) Sample a random minibatch of
\((S(j),A(j),R(j),S(j+1))\)  from \(D\) .
(4) Calculate the target Q-value \(Q_{target}(j)\)  using
the target Q-network: \(Q_{target}(j)=R(j)+\) 
\(\gamma max_{a^{\prime }} Q(s^{\prime },a^{\prime },\theta ^{\prime })\) .
(5) Every \(G\)  steps, use the loss function:
\(L(\theta )=[Q_{target}(j)-Q(s,a,\theta )]^2\) 
to update the target \(Q\) -network.
