After pre-training, we deploy the agent in the complex games, and train the action selector through RL.
We freeze the pre-trained modules, as no human labeled data will be obtained in this phase.
At each time step, we use the task selector and the action validator to produce \(\mathcal {T}_t\)  and \(\mathcal {A}_t\) , respectively.
We keep using the same subtask \(T\)  over time until it is not included in \(\mathcal {T}_t\) , as we do not want the agent to switch subtasks too frequently.
The agent can simply treat \(T_t\)  as the additional observation of \(o_t\) .
If we do not limit the use of human knowledge in this phase, we can also treat \(T_t\)  as a goal with either hand-crafted [1]} or learnt reward function [2]}.
Arbitrary methods can be used for optimizing [3]}, [4]}.
<TABLE>