Deep deterministic policy gradient (DDPG) [1]} is an actor-critic method which parameterizes the deterministic policy (`actor') and the state-action value function (`critic') as neural networks. The critic is updated by minizing the mean squared TD error \(L(\phi , D)\)  in Eq. , where \(D\)  is a set of collected transitions. The actor is updated by solving the Eq. REF .
\(\max _{\theta } \mathbb {E}_{s_t\sim D} \Big [Q_{\phi }(s_t,\mu _{\theta }(s_t))\Big ]\) 
