where \(y_k(i)=r_k(i)+\gamma \tilde{Q}(\mathbf {s}_k(i+1),\tilde{\pi }(\mathbf {s}_k(i+1),\tilde{\mathbf {\phi }}),\tilde{\mathbf {\varphi }})\) , \(\mathbf {\varphi }\)  is the weights of the critic network. Minimization of (REF ) can be solved by using the gradient descent technique as \(\mathbf {\varphi }\leftarrow \mathbf {\varphi }-\beta _c\nabla _{\mathbf {\varphi }}L_k(\mathbf {\varphi })\) , where \(\beta _c\)  is the learning rate. According to the DPG theorem [1]}, the actor network updates its weights in the direction of getting larger cumulative discounted reward. Thus, the weights of the actor network \(\mathbf {\phi }\)  can be updated by minimizing the following loss function
\(L_k^{A}(\mathbf {\phi }) = -Q(\mathbf {s}_k(i), \pi (\mathbf {s}_k(i), \mathbf {\phi }),\mathbf {\varphi }),\forall k.\) 
