One of the classic modern RL algorithms is the Deep Deterministic Policy Gradient algorithm [1]}. It utilizes actor-critic architecture. Critic \(C\)  is a neural network that estimates Q-function (eq. REF ) of a specific observation-action pair. It is trained by gradient descent with a loss function built upon time-difference error (TD-error): \(L = \mathbb {E}_{(o, a, o^{\prime }, r, d} (C(o, a) - (r + \gamma \cdot (1-d) \cdot Q_{A}(o^{\prime }, A(o^{\prime })))^2\) . In practice, \(Q_{A}\)  is usually replaced by value yield by a target critic network that slowly updates to match the critic network. Actor \(A\)  is a neural network that maps observations into an action space. It is trained by a stochastic gradient descent algorithm with a loss function defined as \(L = - \mathbb {E}_{o} C(o, A(o))\)  in order to maximize an expectation of Q-function produced by the critic.
