I am trying to understand the training phase of the tutorial Using Keras and Deep Deterministic Policy Gradient to play TORCS (mirror, code) by Ben Lau published on October 11, 2016.The tutorial says:Then the actor policy is updated using the sampled policy gradient:  $$\nabla_\theta J = \frac{\partial Q^{\theta}(s,a)}{\partial a}\frac{\partial \mu(s|\theta)}{\partial \theta}$$which in the code corresponds to: actor.train(states, grads).In the actor.train() method, I fail to see where $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ gets multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$.I did read:self.params_grad = tf.gradients(self.model.output, self.weights, -self.action_gradient)where self.action_gradient corresponds to $\frac{\partial Q^{\theta}(s,a)}{\partial a}$, and tf.gradients(self.model.output, self.weights) corresponds to $\frac{\partial \mu(s|\theta)}{\partial \theta}$, but I see no  multiplication.Where does $\frac{\partial Q^{\theta}(s,a)}{\partial a}$ get multiplied by $\frac{\partial \mu(s|\theta)}{\partial \theta}$?
Understanding the training phase of the tutorial Using Keras and Deep Deterministic Policy Gradient to play TORCS tutorial
neural network;deep learning;tensorflow;reinforcement learning;training