where \(Q_{{\mathbf {\theta }}^\prime }\)  is the target network [1]}, [2]}.
The actor \(\pi _{\mathbf {\phi }}\)  is trained in the policy improvement step by
\(\textstyle \arg \max _{\mathbf {\phi }}\mathbb {E}_{s \sim {\mathcal {D}}_{\mathrm {env}},\, a \sim \pi _{{\mathbf {\phi }}}(\cdot \,|\,s)}\left[Q_{{\mathbf {\theta }}}\left(s, a\right)\right],\) 
