More recent work seeks to accelerate training with the Adam algorithm, which applies momentum on a per-parameter basis and automatically adapts step size subject to a user-specified maximum (Kingma and Ba, 2015) . While this can lead to much faster convergence, the resulting models are shown to slightly underperform compared to annealing SGD<cite> (Wu et al., 2016)</cite> .