Since the introduction of the Adam algorithm, many variations of the algorithm have been proposed (see Table 2 in the survey paper [1]}). Although many of them might have marginal benefits in practice [1]}, a few of them seem to be able to really provide an improvement of the training process when compared to Adam. Specifically, the work [3]} demonstrates that \(L_{2}\)  regularization and weight decay are not equivalent for adaptive gradient methods. Based on their analysis, they propose the AdamW algorithm, which decouples the weight decay from the gradient-based update. The AdaBelief algorithm [4]} modifies the calculation of the scaling term \(v_t\)  (corresponding to the denominator in the weight update step of the Adam algorithm) in a way which takes into account the "belief" in the gradient. Experiments with a variety of models (CNN, LSTM, GAN) show that this modification improves the performance of the training. Furthermore, recently the AdaMomentum algorithm [5]} was proposed. It also modifies the calculation of the scaling term \(v_t\)  by replacing the gradient with its exponential moving average (EMA). They also propose to move the addition of the constant \(\epsilon \)  (to prevent for the division by zero) to a different term and justify why this is advantageous.
