For the network training at the first phase we use heuristic algorithm described above. In this case it specifies policy \(\pi ^{h}\)  and generates demonstrations for training. The difference of these demonstrations from [1]} is it includes some additional parameters such as, reward data and coordinates of the second point \({(s_t, a_t, r_t, x_2, y_2)}_{t=0,1,2,...}\) . Our pre-training algorithm is based on DDPGfD [2]}. In their work they consider solving robotic problems with sparse rewards using pre-training with expert demonstrations. There are two key differences between our work and [2]}: First, in our case we get dense rewards for our actions, Hence, we omit some losses in the objective function. Second, we use heuristic algorithm to generate demonstrations which might not be always optimal solution for the problem. Therefore, additionally we add Behavioral Cloning (BC) loss for training the actor with Q-filter [4]}. The difference here is we use heuristic policy generator. This is to eliminate non-optimal learning trajectories.
\(L_{BC} = \sum _{i=1}^{n_{D}} {\pi (s_i|\theta _{\pi }) - \pi ^{h}(s_i)}^2 {1}_{Q(s_i , \pi ^{h}(s_i) )> Q(s_i , \pi (s_{i}))}\) 
