The adversarial training is done as in [1]}. One round of training constitutes one epoch (700 iterations) of training \(G\) , followed by two epochs (1400 iterations) of training \(D\) . At each iteration in the training of \(G\) , the policy gradients based loss is used to update the parameters of \(\hat{G}_\theta \)  and adversarial loss is used to update the parameters of \(\tilde{G}_\gamma \) . Firstly, a sequence \(Y_{1:T} = (y_1, ... , y_T )\)  is generated from \(G\) . The action value function \(Q(a = y_t;s=Y_{1:t-1})\)  is then calculated for \(1 \le t \le T\)  by using the Monte Carlo rollout Equation REF , only for the pen-states. The parameters of \(\hat{G}_\theta \)  are updated by policy gradient Equation REF . For finding the adversarial loss, a sequence \((S_0, ... ,S_{N_{max} - 1})\)  of length \(N_{max}\)  is given to \(G\) . The generated sequence is then prepended with \(S_0\)  and a batch containing such sequences is given to \(D\)  for its decision. Based on this decision, the weights of \(\tilde{G}_\gamma \)  are updated. The training of \(D\)  is similar to its pre-training.
