We employ a state-of-the-art RL algorithm called policy-gradient actor-critic [1]} to train both DNNs. “Actor” refers to the component that learns the parameterized policy \(\pi _\theta (a_t | s_t)\) , and “critic” is the component that learns the parameterized value function \(v_w(s_t)\)  in order to evaluate the actor's policy, i.e., to “criticize” it. The actor updates the policy parameters \(\theta \)  by applying gradient ascent (with step-size \(\alpha \) ) in the direction the critic indicates as “good”. The critic evaluates the actor's policy by measuring \(\delta =G_t-v_w(s_t)\) , i.e., the error between \(v_w(s_t)\)  that it estimated and the actual return received from the environment. The parameters \(w\)  of the critic are also trained in the direction indicated by \(\alpha \times \delta \) .
