We propose NovoGrad, an adaptive stochastic gradient descent method with layer-wise gradient normalization and decoupled weight decay. In our experiments on neural networks for image classification, speech recognition, machine translation, and language modeling, it performs on par or better than well tuned SGD with momentum and Adam/AdamW. 
 Additionally, NovoGrad (1) is robust to the choice of learning rate and weight initialization, (2) works well in a large batch setting, and (3) has two times smaller memory footprint than Adam. The most popular algorithms for training Neural Networks (NNs) are Stochastic Gradient Descent (SGD) with momentum (Polyak, 1964; Sutskever et al., 2013) and Adam (Kingma & Ba, 2015) . SGD with momentum is the preferred algorithm for computer vision, while Adam is the most commonly used for natural language processing (NLP) and speech problems. Compared to SGD, Adam is perceived as safer and more robust to weight initialization and learning rate. However, Adam has certain drawbacks. First, as noted in the original paper (Kingma & Ba, 2015) , the second moment can vanish or explode, especially during the initial phase of training. To alleviate this problem, a learning rate (LR) warmup (Goyal et al., 2017 ) is typically used. Adam often leads to solutions that generalize worse than SGD (Wilson et al., 2017) , and to improve Adam regularization, Loshchilov & Hutter (2019) proposed AdamW with decoupled weight decay. Our motivation for this work was to find an algorithm which: (1) performs equally well for image classification, speech recognition, machine translation, and language modeling, (2) is robust to learning rate and weight initialization, (3) has strong regularization properties. We start with Adam, and then (1) replace the element-wise second moment with the layer-wise moment, (2) compute the first moment using gradients normalized by layer-wise second moment, (3) and decouple weight decay (similar to AdamW) from normalized gradients. The resulting algorithm, NovoGrad, combines SGD's and Adam's strengths. We applied NovoGrad to a variety of large scale problems -image classification, neural machine translation, language modeling, and speech recognition -and found that in all cases, it performs as well or better than Adam/AdamW and SGD with momentum.
NovoGrad -  an adaptive SGD method with layer-wise gradient normalization and decoupled weight decay. 
{"src": "scitldr-AIC_train", "paper_id": "BJepq2VtDB"}