Algorithm 1 is a modified version of DDPG. For the purpose of comparison, we keep most of the symbols and statements intact from the original paper. Please refer to [1]} for the detail of the algorithm. The difference from DDPG is that action \(a\)  in DDPG of learning process is replaced by probability vector \(p\) . Action \(a\)  in PSADPG sampled from \(p\)  is only used to interact with environment. Probability vector \(p\)  is the output of the softmax layer of actor network. To improve the efficiency of experience replay, we store a unit vector instead of the probability vector generated by the policy. The only non-zero element of this unit vector corresponds to the action \(a_t\) .
