We chose the following hyperparameters to train the systems.
The embedding and recurrent layers have a dimensionality of 620 and 1000 respectively.
We use a minibatch size of 80 sentences trained with Adadelta algorithm.
The norm of the gradient is clipped to be no more than 1 [1]} and the weights are initialized with Xavier [2]}.
The validations start at the second epoch and are performed every 5000 updates.
Early stopping is based on BLEU with a patience set to 10 (early stopping occurs after 10 evaluations without improvement in BLEU).
The vocabulary size of the source languages is set to 30K.
We varied the output layer size from 5K to 30K in order to simulate different levels of out of domain data.
Once the model is trained, we set the beam size to 12 (as this is the standard value for NMT, [3]}) when translating the development corpus.
