Using two replay buffers: We keep two replay buffers \(R_D\)  for demonstration data and \(R_A\)  for agent's self exploration data. During the training process in each step two minibatches sampled from each of the replay buffers for the network training. Demonstration data in our work is generated by heuristic algorithm and is different from the robotic control tasks [1]}, [2]} where demonstration data is gathered using real robots and human implication. Therefore, in robotic learning tasks it is meaningful to keep agent stick on demonstration data to avoid unrealistic scenarios. In this work, we use only efficient demonstration data.
Agent training process is given in the following Algorithm.1.
