In the integration ( distillation ) step , we are interested in combining all past skills ( π 0 , . . . , π i ) with the newly acquired skill π i+1 . Traditional approaches have used policy regression where data is generated by collecting trajectories of the expert policy on a task . Training the student on these trajectories does not always result in robust behaviour . This poor behaviour is caused by the student experiencing a different distribution of trajectories than the expert during evaluation . To compensate for this distribution difference , portions of the trajectories should be generated by the student . This allows the expert to suggest behaviour that will pull the state distribution of the student closer to the expert 's . This is a common problem in learning a model to reproduce a given distribution of trajectories ( Ross et al . , 2010;Bengio et al . , 2015;Martinez et al . , 2017;Lamb et al . , 2016 ) . We use a method similar to the DAGGER algorithm ( Ross et al . , 2010 ) which is useful for distilling policies ( Parisotto et al . , 2015 ) . See Appendix : 8.2.1 for more details . As our RL algorithm is an actor - critic method , we also perform regression on the critic by fitting both in the same step .