In practice, both the updates for A2C and BC are computed with RAdam and a batch size of 5.
Two hyper-parameters, \(C_{BC}\) , \(C_{RL}\) , represent how many gradient updates are performed with a method over the other control how dominating is one update over the other.
The simple strategy as presented in Algorithm REF , can be implemented in the trainer functions, also for distributed environments such as SEED RL [1]}.
At the beginning of the training of the policy \(\pi _\theta ^{MB}\) , they are equal, meaning that the two methods have the same importance over the learned policy.
In our case we, we set \(C_{BC} = C_{RL} = 40\) , such that, with a batch size of 5, 200 state-action pairs are used for each method.
Changing the two hyper-parameters allows for controlling how much the bot sticks to the BC policy and how much it drifts away towards the behaviour purely learned using A2C.
After 12M of steps executed using A2C, we set \(C_{BC}=0\) , meaning that no BC iterations are performed.
<FIGURE><FIGURE>