More recent work seeks to accelerate training with the Adam algorithm, which applies momentum on a per-parameter basis and automatically adapts step size subject to a user-specified maximum (Kingma and Ba, 2015) . While this can lead to much faster convergence, the resulting models are shown to slightly underperform compared to annealing SGD<cite> (Wu et al., 2016)</cite> . However, Adam's speed and reputation of generally being "good enough" have made it a popular choice for researchers and NMT toolkit authors 6 (Arthur et al., 2016; Lee et al., 2016; Britz et al., 2017; Sennrich et al., 2017) .