During training, we use Adadelta [1]}, a minibatch size of 80, and reshuffle the training set between epochs.
We train a network for approximately 7 days, then take the last 4 saved models (models being saved every 12 hours), and continue training each with a fixed embedding layer (as suggested by [2]}) for 12 hours.
We perform two independent training runs for each models, once with cut-off for gradient clipping [3]} of 5.0, once with a cut-off of 1.0 – the latter produced better single models for most settings.
We report results of the system that performed best on our development set (newstest2013), and of an ensemble of all 8 models.
