Critic baseline   
Figure REF  illustrates that, for identical models, the critic baseline [1]}, [2]} is unable to match the performance of the rollout baseline [3]} under both greedy and beam search settings.
We did not explore separately tuning learning rates and hyperparameters for the critic network, opting to use the same settings as those for the actor.
In general, getting actor-critic methods to work seems to require more parameter tuning than the rollout baseline.
