Momentum Doesn't Change the Implicit Bias

The momentum acceleration technique is widely adopted in many optimization algorithms. However, the theoretical understanding on how the momentum affects the generalization performance of the optimization algorithms is still unknown. In this paper, we answer this question through analyzing the implicit bias of momentum-based optimization. We prove that both SGD with momentum and Adam converge to the L2 max-margin solution for exponential-tailed loss, which is the same as vanilla gradient descent. That means, these optimizers with momentum acceleration still converge to a model with low complexity, which provides guarantees on their generalization. Technically, to overcome the difficulty brought by the error accumulation in analyzing the momentum, we construct new Lyapunov functions as a tool to analyze the gap between the model parameter and the max-margin solution.


INTRODUCTION
It is widely believed that the optimizers have implicit bias in terms of selecting output parameters among all the local minima on the landscape. The pioneer work (Soudry et al., 2018) shows that gradient descent would converge to the L 2 max-margin solution for the linear classification task with exponetial-tailed loss, which mirrors its good generalization property in practice. Since then, many efforts have been taken on analyzing the implicit bias of various local-search optimizers, including stochastic gradient descent (Nacson et al., 2019), AdaGrad (Qian & Qian, 2019) and optimizers for homogeneous neural networks (Lyu & Li, 2019;Wang et al., 2021;Qian & Qian, 2019).
However, though the momentum acceleration technique is widely adopted in the optimization algorithms in both convex and non-convex learning tasks (Sutskever et al., 2013;Vaswani et al., 2017;Tan & Le, 2019), the understanding on how the momentum would affect generalization performance of the optimization algorithms is still unclear. A natural question is:

Can we theoretically analyze the implicit bias of momentum-based optimizers?
In this paper, we take the first step to analyze the convergence of momentum based optimizers and unveil their implicit bias. Specifically, we study the classification problem with linear model and exponential-tailed loss using Stochastic Gradient Descent with Momentum (SGDM) and Adam optimizers. We consider the optimizers with constant learning rate and constant momentum hyperparameters, which are widely adopted in practice, e.g., the default setting in popular machine learning frameworks (Paszke et al., 2019) and in experiments (Xie et al., 2017). We note that Gradient Descent with Momentum (GDM) can be viewed as a special case of SGDM, and naturally share the properties for SGDM. Our main results are summarized in Theorem 1.
Theorem 1 (informal). With linear separable dataset S, for SGDM and Adam, the loss converges to 0 with rate O( 1 t ) where t is the number of iterations, the parameter norm diverges to infinity, and direction of parameters converges to the direction of the L 2 max-margin solution.
Theorem 1 states SGDM converges to the L 2 max-margin solution, which is the same as SGD, indicating that momentum does not affect the convergent direction. The good generalization behavior of the output parameters of SGDM is well validated as the margin of a classifier is positively correlated with its generalization error (Jiang et al., 2019). This is supported by existing experimental observations (c.f. Figure 1, (Soudry et al., 2018) and Figure 2, (Nacson et al., 2019)). Similar claims hold for Adam, which is also well supported by empirical results in Wang et al. (2021).
Our contributions are significant in terms of the following aspects: • We establish the implicit bias of the momentum based optimizers, an open problem since the initial work Soudry et al. (2018). The momentum based optimizers are widely used in practice and our theoretical characterization deepens the understanding on their generalization property, which is important by its own.
• Technically, we propose a new Lyapunov function to analyze the convergence of SGDM, which helps to bound the sum of squared gradients along the training trajectory. Compared to the usual one, the new Lyapunov function depends on a middle variable of an alternative update rule of SGDM, which helps to capture the historical dependence in the momentum update. To our knowledge, such a technique has not been exploited ever before and can be of independent interest for convergence analysis of momentum-based optimizers. We then construct a new Lyapunov function to bound the difference of learned parameters and the scaled max-margin solution, which finally leads to the direction convergence. This Lyapunov function provides a direct way to establish the convergence to the desired direction.
Organization of This Paper. Section 2 collects further related works on the implicit bias of first order optimizers and convergence of momentum-based optimizers. Section 3 shows basic settings and assumptions which will be used throughout this paper. Section 4 studies the implicit bias of GDM as a warm up, while Section 5 and Section 6 explore respectively the implicit bias of SGDM and Adam. Discussions of these results are put in Section 7.

FURTHER RELATED WORKS
Implicit Bias of First-order Optimization Methods. Soudry et al. (2018) prove that gradient descent on linear classification problem with exponential-tailed loss converges to the direction of the max L 2 margin solution of the corresponding hard-margin Support Vector Machine. Nacson et al.
(2019) extend the results in (Soudry et al., 2018) to the stochastic case, proving that the convergent direction of SGD is the same as GD almost surely. Qian & Qian (2019) go beyond the vanilla gradient descent methods and consider the AdaGrad optimizer instead. They prove that the convergent direction of AdaGrad has a dependency on the optimizing trajectory, which varies according to the initialization.  propose a primal-dual analysis framework for the linear classification models, and prove a faster convergent rate of the margin by increasing the learning rate according to the loss. Based on ,  design another algorithm with an even faster convergent rate of margin by applying the Nesterov's Acceleration Method on the dual space. However, the corresponding form of the algorithm on the primal space is no longer a Nesterov's Acceleration Method nor GDM, which is significantly different from our settings.
On the other hand, there is another line of work trying to extend the result in the linear case to deep neural networks. Ji & Telgarsky (2018); Gunasekar et al. (2018) study the deep linear network and Soudry et al. (2018) study the two-layer neural network with ReLU activation. Lyu & Li (2019) propose a framework to analyze the asymptotic direction of GD on homogeneous neural networks, proving that given there exists a time the network achieves 100% training accuracy, GD will converge to some KKT point of the L 2 max-margin problem. Wang et al. (2021) extend the framework of Lyu & Li (2019) to adaptive optimizers, and prove RMSProp and Adam without momentum have the same convergent direction as GD, while AdaGrad doesn't. The results (Lyu & Li, 2019;Wang et al., 2021) indicate that results in the linear model can be extended to deep homogeneous neural networks, and suggest that the linear model is a proper start point to study the implicit bias.
Convergence of Momentum-Based Optimization Methods. For convex optimization problems, the convergence rate of Nesterov's Acceleration Method has been proved in Nesterov (1983). In contrast, although GDM (Polyak's Heavy-Ball Method) was proposed in (Polyak, 1964) prior to the Nesterov's Acceleration Method, the convergence of GDM on convex loss with Lipschitz gradient was not solved until Ghadimi et al. (2015) provides an ergodic convergent result for GDM, i.e., the convergent result for the running average of the iterates. However, the ergodic result is undesired under many learning scenarios, e.g., in classification tasks, the optimization algorithms usually output the parameters of the last step. To the best of our knowledge, there are only two works on the non-ergodic analysis of GDM: Sun et al. (2019) and Tao et al. (2021). Sun et al. (2019) prove that if the training loss is coercive (the training loss goes to infinity whenever parameter norm goes to infinity), convex, and globally smooth, then, with constant momentum hyper-parameter, the training loss converges to minima with rate O(t −1 ). Tao et al. (2021) analyze a case when momentum coefficient increases to 1 and the gradient is bounded all over the parameter space, showing SGDM can improve the convergence rate of SGD by a factor log(t).
However, there is no existing work on the implicit bias of momentum-based optimizers for the classification problem, which is first analyzed by this paper.

PRELIMINARIES
In this paper, we focus on the linear model with exponential-tailed loss. We first derive the results for binary classification, then we show that the methodology can be easily extended to the multi-class classification problem.
Problem setting. The dataset used for training is defined as , where x i ∈ R d is the i-th feature, and y i ∈ R is the i-th label (i = 1, 2, · · · , N ). We will use the linear model to fit the label: for any feature x ∈ R d and parameter w ∈ R d , the prediction is given by w, x .
For binary classification, given any data z i = (x i , y i ) ∈ S, the individual loss for parameter w is given as ℓ(y i w, x i ). A common setting is to ensemble the feature and label together asx i = y i x i (there is a mapping T : (x, y) → yx, andx i is T ((x i , y i ))). The individual loss can be rewritten asl The optimization target is defined as the averaged loss: Without loss of generality, we consider the case with normalized data 1 , that is, Optimizer. Here we will introduce the update rules of SGDM and Adam. SGDM can be viewed as a stochastic version of GDM by randomly choosing a subset of the dataset to update. Specifically, the update rule of SGDM is given as where B(t) is a subset of S with size b which is sampled independently and uniformly with replace- We also define F t as the sub-sigma field such that {w(t)} ∞ t=1 is adapted with respect to Filtration {F t } ∞ t=1 . The Adam can be viewed as a variant of GDM in which the preconditioner is adopted, whose form is characterized as follows: where 1 √ν (t−1)+ε1 d ) is called the preconditioner at step t.
1 The proof can be naturally applied to unnormalized data by letting ℓ(x) = ℓ(maxx ∈T (S) x · x) and Assumptions: The analysis of this paper are based on three assumptions, respectively on the separability of the dataset, the individual loss behaviour at the tail , and the smoothness of the individual loss. We list them as follows: Assumption 1 (Linearly Separable Dataset). There exists one parameter w ∈ R d , such that Assumption 2 (Exponential-tailed Loss). The individual loss ℓ is exponential-tailed, i.e., • Differentiable and monotonically decreasing to zero, with its derivative also converging to zero, i.e., lim x→∞ ℓ(x) = lim x→∞ ℓ ′ (x) = 0, and ℓ ′ (x) > 0 ∀x; • Close to exponential loss when x is large enough, i.e., there exist positive constants c, a, K, µ + , µ − , x + , x − and x 0 , such that, We provide explanations of these three assumptions respectively. Based on Assumption 1, we can formally define the margin and the maximum margin solution of an optimization problem, which is introduced in Definition 1 Definition 1. Let the margin γ(w) of parameter w defined as the lowest score of the prediction of w over the dataset S, i.e., γ(w) = minx i∈T (S) w,x i . By Assumption 1 and the positive homogeneous of γ, γ(ŵ γ(ŵ) ) = 1, and thus we define the maximum margin solutionŵ and the margin γ of the dataset S as follows: Since · 2 is strongly convex and set {w : γ(w) ≥ 1} is convex,ŵ is uniquely defined.
Assumption 2 constraints the loss to be exponential-tailed, which is satisfied by many popular choices of ℓ, including the (ℓ exp (x) = e −x ) and the logistic loss (ℓ log (x) = log(1 + e −x )). Also, as c and a can be respectively absorbed by resetting the learning rate and data as η = cη and x i = ax i , without loss of generality, in this paper we only analyze the case that c = a = 1.
The globally smooth assumption (Assumption 3. (S)) is strictly stronger than the locally smooth assumption (Assumption 3. (D)). One can easily verify that both the exponential loss and the logistic loss meet Assumption 3. (D), and the logistic loss also meets Assumption 3. (S).

WARM UP: IMPLICIT BIAS OF GDM
In this section, we study the implicit bias of GDM as a warm up. The update rule of GDM is which is a determined case of SGDM, with batch size b = N and thus without randomness. Although the analysis of GDM is essentially easier than that of SGDM, it helps us gain a better understanding of this problem, and helps us to demonstrate several techniques which will be applied latter. The formal theorem of the implicit bias of GDM is as follows: Theorem 2. Let Assumptions 1, 2, and 3. (D) hold. Let β ∈ [0, 1) and η < 2 . Then, for almost every data set S, with arbitrary initialization point w(1), GDM (Eq. (5)) satisfies that w(t) − log(t)ŵ is bounded as t → ∞, and lim t→∞ Before we move on to the proof, we would like to provide some explanations of the results in Theorem 2. To begin with, Theorem 2 adopts an constant momentum hyper-parameter, which agrees with the practice use (e.g., β is set to be 0.9 (Xie et al., 2017)). Also, Theorem 2 puts no restriction on the range of β, which allows wider choices of hyper-parameter tuning. Furthermore, Theorem 2 shows the implicit bias of GDM agrees with GD in linear classification with exponential-tailed loss (c.f. Soudry et al. (2018) for results on GD), and this consistency can be verified by existing results (c.f. Section 7 for detailed discussions).
We then present a proof sketch of Theorem 2, which is divided into two parts: we first prove that the sum of squared gradients is bounded, which indicates both the loss and the norm of gradient converge to 0; we then show the difference between w(t) and log(t)ŵ is bounded, and therefore, the direction ofŵ dominates as t → ∞.
Stage I: Bound the sum of squared gradients. The core of Stage I is to select a proper Lyapunov function ξ(t), which is required to correlated with the training loss L and be non-increasing along the optimization trajectory. For GD, since L itself is non-increasing with properly chosen learning rate, we can just pick ξ(t) = L(t). However, as the update of GDM doesn't align with the direction of negative gradient, training loss L(t) in GDM is no longer monotonously decreasing, and the Lyapunov function requires special construction. A choice of ξ(t) is proposed as the follows: Remark 1. Although this Lyapunov function is obtained by (Sun et al., 2019) by directly examining the Taylor's expansion at w(t), the proof here is non-trivial as we only requires the loss to be locally smooth instead of globally smooth in (Sun et al., 2019), and the Taylor's expansion can only be applied to w(t + 1) if it's ensured that all parameters on the line αw(t) + (1 − α)w(t + 1) (α ∈ (0, 1)) have training loss no larger than L(w(0)).
To prove Lemma 1, we define w(t + α) = αw(t) + (1 − α)w(t + 1) for any t ∈ Z + and α ∈ (0, 1), and prove a generalized version of Eq. (6): The proof idea is that as long as Eq. (7) holds for all time in [1, t + α), the training loss across [1, t + α] will be smaller than L(w(1)), and the strict inequality in Eq. (7) holds. Consequently, there exists some small enough positive real ε, such that Eq. (7) holds for all time in [1, t+α+ε), and we are able to extend the feasible set where Eq. (7) holds. The formal description of the generalized lemma and its corresponding proof is deferred to Appendix B.1.1.
However, it no longer holds for GDM, as w(t + 1) − w(t) is a exponentially-decayed sum of gradients at w(s), s ≤ t. To handle this problem, we propose a novel Lyapunov's function for the direction convergence, concluded as the following Lemma.

Lemma 2. Let all conditions in Theorem 2 hold. Then, r(t) is bounded if and only if the function
) is upper bounded.
The first claim in Lemma 2 provides an alternative approach to verify that r(t) is bounded, while the second claim shows this approach can be fulfilled. The motivation of this lemma is to construct a easy-to-verify criterion (i.e., g(t) is upper bounded) of r(t) is upper bounded. To demonstrate how g(t) is constructed intuitively, we consider the continuous dynamics approximation of GDM, i.e., We then calculate the 1 2 r(t) 2 − 1 2 r(1) 2 by applying the integral of the dynamics as where the last equation is due to Integration by part. We ds , dw(s) ds ds) + 1 2 r(t) 2 , and it can be verified the derivative ofg(t) is r(t), − η 1−β ∇L(w(t)) − 1 tŵ . By replacing dw(t) dt as w(t) − w(t − 1), and dr(t) dt as r(t) − r(t − 1) ing(t), we obtain g(t), and g(t + 1) − g(t) has the form r(t), − η 1−β ∇L(w(t)) − log t+1 tŵ , which can be analyzed following the similar routine as (Soudry et al., 2018). The proof of the second claim is completed.

TACKLE THE DIFFICULTY BROUGHT BY RANDOM SAMPLING
In this section, we analyze the implicit bias of SGDM. The randomness introduced by random sampling makes the analysis of GDM no longer work for SGDM. As an example, if we want to follow the same routine of the GDM case to show L(w(t)) + β 2η E w(t) − w(t − 1) 2 is a Lyapunov function of SGDM, we can only have As the squared first momentum is smaller than the second momentum, EL(w(t)) + β 2η E w(t) − w(t − 1) 2 may no longer be monotonously decreasing for every β ∈ [0, 1). Furthermore, if the margin γ is small, β will be required to be upper bounded by a small number, which contradicts to the relatively large choice of β in practice, e.g. 0.9 (We defer a detailed discussion to Appendix B.2.3). To tackle this problem, we propose a new Lyapunov function, with which we derive the following theorem for the implicit bias of SGDM: Theorem 3. Let Assumption 1, 2, and 3. (S) hold. Let β ∈ [0, 1) and η < min{ 1−β 2H 2 N 3 β 2 }. Then, for almost every data set S, with arbitrary initialization point w(1), SGDM (Eq. (1)) satisfies w(t) − log(t)ŵ is bounded as t → ∞ and lim t→∞ w(t) w(t) =ŵ ŵ , almost surely (a.s.).
To the best of our knowledge, this is the first convergence analysis of SGDM with no restriction on the gradient norm and with constant learning rates. Also, as both 1−β are monotonously increasing with respect to batch size b, Theorem 3 also sheds light on the learning rate tuning, i.e., the larger the batch size is, the larger the learning rate is. Furthermore, similar to the GDM case, Theorem 3 shows the implicit bias of SGDM under this setting is consistent with SGD (c.f. (Nacson et al., 2019) for the implicit bias of SGD). This matches the observations in practice (c.f. Section 7 for details). Remark 2. In practice, mini-batch SGDM are more widely adopted, whose update rule differs from SGDM only by the way obtaining B(t). Specifically, within the T -th epoch E(T ) = {KT + 1, · · · , KT +T } (K is the length of one epoch), {B(t)} t∈E(T ) is a randomly and uniformly partition of S. One may wonder whether the same result hold for mini-batch SGDM. The answer is "Yes", with the a.s. condition removed in this case. We defer the detailed description of the corresponding theorem together with the proof to Appendix D.1.
The proof sketch follows the same framework as that for GDM by dividing the proof into two stages. However, the implementations in both stages differ, among which the proof of Stage I is significantly distinctive, as a new Lyapunov function is proposed.
By a simple rearrangement of the second equation, we have u(t+1) = w(t)+ 1 1−β (w(t+1)−w(t)), which differs from w(t + 1) only by a larger step size updated from w(t). The following lemma then indicates that EL(u(t)) is a proper selection of Lyapunov function. Lemma 4. Let all conditions in Theorem 3 hold. Then, we have EL(u(t)) may be not monotonously decreasing. However, the upper bound of EL(u(t)), i.e., L(u(1)) − t s=1η 4 'E ∇L(w(s)) 2 , is monotonously decreasing byη 4 E ∇L(w(t)) 2 at step t, and leads to the sum of expected squared gradients being finite along the trajectory (which further indicates the sum of squared gradients is finite, a.s.).
Stage II. Bound the difference between w(t) and log(t)ŵ. The Lyapunov function used by Stage II of SGDM is the same as the g(t) in the GDM case. As we have proved the sum of squared gradient along the trajectory is bounded, one can easily obtain that ∞ t=1 w(t + 1) − w(t) 2 is bounded, and r(t) being bounded is still equivalent to that g(t) is upper bounded. However, when it comes to the detailed calculation of analyzing g(t + 1) − g(t), it differs from the GDM case due to the random subset. We defer the proof to Appendix B.2.2.

ANALYZE THE EFFECT OF PRECONDITIONERS
When it comes to the analysis of Adam, the effect of the preconditioner should be taken into consideration when designing the corresponding Lyapunov functions. To tackle this problem, we incorporate the preconditioner into the Lyapunov function, and obtain the implicit bias of Adam as follows: Theorem 4. Let Assumption 1, 2, and 3. (D) hold. Let 1 > β 2 > β 4 1 ≥ 0, and the learning rate η is a small enough constant (The upper bound of learning rate is complex, and we defer it to Appendix C.1). Then, for almost every data set S, with arbitrary initialization point w(1), Adam (Eq. (2)) satisfies that w(t) − log(t)ŵ is bounded as t → ∞, and lim t→∞ Remark 3. Existing literature usually assume a time-decaying hyperparameter choice of β 1 or β 2 (c.f., (Kingma & Ba, 2014;Chen et al., 2018)). To the best of our knowledge, the only work analyzing Adam with constant β 1 and β 2 is (Reddi et al., 2019), which assumes β 2 > β 2 1 (stronger than our assumption). Our result indicates that the ranges of β 1 and β 2 can be broader in hyperparameter selection.
We still start the proof sketch by proving the sum of squared gradients is finite in Stage I. Compared to GDM, the difference is that in Stage I, we will also prove the loss converges to 0 with rate Θ(t −1 ). This property will be used in Stage II to analyze the effect of preconditioner on implicit bias.
Stage I: Bound the sum of squared gradients. The next lemma can be viewed as an extension of Lemma 1 by taking the preconditioner into consideration, and characterizes the one-step loss update in Adam. Lemma 5. Let all conditions in Theorem 4 hold. Then, for any t ≥ 1, The difference between the proof of Lemma 5 and Lemma 1 is that we need to handle the gap between the preconditioners at step t and step t + 1, this leads to a amplifying factor 4 , which, however, is smaller than the shrinking factor ) introduced by the gap between the coefficients of momentumm(t) and m(t − 1). As t → ∞, both β t 1 → 0 and (cβ 1 ) t → 0, and we can obtain that ξ(t) is a Lyapunov function. Based on Lemma 5, we can prove the following asymptotic rate of L. Lemma 6. Let all conditions in Theorem 4 hold. Then, The proof idea of Lemma 6 is that by the exponential-tailed property, when time is large enough, the training loss can be bounded by the gradient, and further bounded by Stage II. Bound the difference between w(t) and log(t)ŵ. The Lyapunov function for the direction difference is also modified according to the preconditioner, as introduced in Lemma 7:

Lemma 7. Let all conditions in Theorem 4 hold. Then, r(t) is bounded if and only if g(t) is upper bounded, where g(t) is defined as follows.
The first claim of Lemma 7 is similar to that in GDM and SGDM case. However, when we come to the second claim, simple calculation of g(t + 1) − g(t) leads to This is where we need Lemma 6, which bounds the gap between ε1 d +ν(t) and can be tackled by following the same routine as the GDM/SGDM case. Consequently, the proof of Lemma 7 is completed.

DISCUSSIONS
Extension to the Multi-Class Classification Problem. As mentioned in the introduction, despite all the previous analyses are aimed at the binary classification problem, they can be naturally extended to the analyses multi-class classification problem. Specifically, in the linear multi-class classification problem, for any (x, y) ∈ R dX × {1, · · · , C} in the sample set S, the (individual) logistic loss with parameter W ∈ R C×dX is denoted as Correspondingly, dataset S is separable if there exists a parameter W , such that ∀(x, y) ∈ S, we have W y, x > W i, x, ∀i = y. The multi-class L 2 max-margin problem is then defined as DenoteŴ as the L 2 max-margin solution, we have (mini-batch) SGDM and Adam still converges to the direction ofŴ , the proof of which is deferred to the Appendix D.2. Theorem 5. For linear multi-class classification problem using logistic loss and almost every separable data, with a small enough learning rate, and 1 > β 2 > β 4 1 ≥ 0 (for Adam), (mini-batch) SGDM and Adam converge to the multi-class L 2 max-margin solution (a.s. for SGDM).  (Lyu & Li, 2019) can be generalized, and successfully extend the Lyapunov function to AdaGrad, RMSProp and Adam (without momentum) on deep homogeneous neural networks. It will be interesting to see whether such a Lyapunov function can be constructed for GDM and Adam in homogeneous neural networks.

CONCLUSION
In this paper, we study the implicit bias of momentum-based optimizers in linear classification with exponential-tailed loss. Our results indicates that for SGD and Adam, adding momentum will not influence the implicit bias, and the direction of parameter converges to the L 2 max-margin solution.
Our theoretical results stands with existing experimental observations.

Supplementary materials for "Momentum Doesn't Change the Implicit Bias"
A PREPARATIONS This section collect definitions and lemmas which will be used throughout the proofs. To start with, we define support vectors and support set, which are two common terms in margin analysis.
Definition 2 (Support vectors and support set).
The following lemma deliversŵ as an linear combination of support vectors.
Lemma 8 (Lemma 12, Soudry et al. (2018)). For almost every datasets S, there exists a unique vector v = (v 1 , · · · , v N ), such thatŵ can be represented aŝ where v satisfies v i = 0 ifx i / ∈ T (S s ), and v i > 0 ifx i ∈ T (S s ). Furthermore, the size ofS s is at most d.
By Lemma 8, we further have the following corollary: Corollary 2. For almost every datasets S, the unique v given by Lemma 8 further satisfies that for any positive constant c, there exists a non-zero vectorŵ, such that,x i ∈ T (S s ), we have Proof. For almost every datasets S, any subsets with size d of S is linearly independent. SinceS s has size no larger than d (by Lemma 8), and Eq. (12) is equivalent to linear equations, the proof is completed.
For the stochastic case, we will also need the following lemma when we calculate the form of parameter at time t.

A.2 PREPARATIONS OF THE OPTIMIZATION ANALYSIS
This section collects technical lemmas which will be used in latter proofs. We begin with a lemma bounding the smooth constants if the loss is bounded.

Lemma 10. If loss ℓ satisfies (D) in Assumption 3, then for any
for any two parameters w 1 and w 2 close enough to w, and thus, which completes the proof that L is locally H s0 smooth at w.
Following the same routine as the locally smooth proof, we complete the second argument.
The proof is completed.
Based on Assumption 2, we also have the following lemma characterizing the relationship between loss ℓ and its derivative ℓ ′ when x is large enough.
Lemma 11. Let loss ℓ satisfy Assumption 2. Then, there exists an large enough x 0 and a positive real K, such that, ∀x > x 0 , we have Proof. By Assumption 2, there exists a large enough x 0 , such that ∀x > x 0 , we have On the other hand, as lim t→∞ ℓ(x) = 0, we have which by Eq. (13) leads to Therefore, setting K = 4 completes the proof.
The following lemma bridges the second moment of ∇L B(t) with its squared first moment.
Lemma 12. Let the dataset S satisfies the separable assumption 1. Let B be a random subset of S with size b sampled independently and uniformly without replacement. Then, at any point w, we have Proof. To start with, notice that Therefore, the first inequality can be directly obtained by Cauchy-Schwartz's inequality. To prove the second inequality, we first calculate the explicit form of ∇L B (w). Therefore, On the other hand, where Eq. (⋆) is due to ∀z ∈ x, −γŵ ≥ γ and ℓ ′ < 0. Therefore, The proof is completed by putting Eqs. (15) and (16) together.
In the following lemma, we show the updates of GDM, Adam, and SGDM are all non-zero. Lemma 13. Regardless of GDM, Adam, or SGDM, the updates of all steps are non-zero, i.e., Proof. We start with the alternative forms of the update rule of GDM, Adam, and SGDM using the gradients along the trajectory respectively. For GDM, by Eq. (5), the update rule can be written as Similarly, the update rule of SGDM can be written as while the update rule of Adam can be given as On the other hand, by the definition of empirical risk L, the gradient of L at point w can be given as By Eq. (20) and Eq. (17), we further have for GDM, By Assumption 1, there exists a non-zero parameterŵ, such that, ŵ,x i > 0, ∀i. Therefore, by executing inner product between Eq. (21) andŵ, we have where Eq. ( * ) is due to ℓ ′ < 0. This complete the proof for GDM.
Similarly, for SGDM, we have which completes the proof of SGDM.
For Adam, we have which completes the proof of Adam.
The proof is completed.

B IMPLICIT BIAS OF GD/SGD WITH MOMENTUM
This section collects the proof of the implicit bias of gradient descent with momentum and stochastic gradient descent with momentum.

B.1 IMPLICIT BIAS OF GD WITH MOMENTUM
This section collects the proof of Theorem 2.

B.1.1 PROOF OF THE SUM OF SQUARED GRADIENTS CONVERGES
To begin with, we will prove the sum of squared norm of gradients along the trajectory is finite for gradient descent with momentum. To see this, we first define the continuous-time update rule as We then prove a generalized case of Lemma 1 for any w(t + α). Lemma 14 (Lemma 1, extended). Let all conditions in Theorem 2 hold. We then have where c is a positive real such that η = 2 1−β Hs 0 c.
We further divide the proof into two cases depending on the value of α * .
By Lemma 10, we have L is H s0 smooth at w(t * ). Therefore, by Taylor's expansion for L at point w(t * ), we have for small enough α > 0 where Eq. ( * ) is due to a simple rearrangement of the update role of gradient descent with momentum (Eq. (5)), i.e., Inequality ( * * ) is due to Cauchy Schwarz's inequality and arithmetic-geometric average inequality, and Inequality (⋄) is due to Eq. (22) indicates Eq. (7) holds at (t * , α) for α > 0 is small enough, which contradicts to α * = 0.
The proof is completed.
By Lemma 1, one can easily obtain the sum of the squared norms of the updates across the trajectory converges.
By triangle inequality, we further have where Eq. (⋆) is due to Cauchy-Schwartz's inequality.
The proof is completed.
By the negative derivative of the loss and the separable data, we can finally prove Corollary 1.
By the exponential-tailed assumption of the loss (Assumption 2), we further have the following corollary.
Consequently, there exists an large enough time t 0 , such that, ∀t > t 0 , ∀i, we have w(t),x i > 0, and

B.1.2 BOUNDING THE ORTHOGONAL PART
To prove Theorem 2, we only need to show w(t) − log(t)ŵ (t ≥ 1) has bounded norm for any iteration t > 0. Letting c = η (1−β)N in Corollary 2, we obtain an constant vectorw satisfying Eq. (12). Define Asw is a constant vector, that w(t) − log(t)ŵ (t ≥ 1) has bounded norm is equivalent to r(t) has bounded norm. Lemma 2 then propose an equivalent proposition of r(t) is bounded, and further prove this proposition is fulfilled. As the proof is rather complex, we separate it into two sub-lemmas. We first prove r(t) is bounded if and only if function g(t) is upper bounded.

Lemma 15 (First argument in Lemma 2). Let all conditions in Theorem 2 hold. Then, r(t) is bounded if and only if function g(t) is upper bounded.
Proof. We start the proof by showing that A 1 (t) By the definition of r(t) , we have Therefore, the absolute value of A 1 (t) can be bounded as where Inequality (⋆) is due to the Inequality of arithmetic and geometric means, and Inequality (•) is due to Corollary 3 and log τ τ −1 = O( 1 τ ).

Therefore, g(t) is upper bounded is then equivalent to
is upper bounded, we will prove r(t) is bounded by reduction to absurdity. By Corollary 3, we have lim t→∞ w(t)− w(t− 1) = 0, and there exists a large enough time T , such that w(t) − w(t − 1) < 1 for any t ≥ T . On the other hand, since r(t) is unbounded from above, there exists an increasing time sequence k i > T , i ∈ Z + , such that lim i→∞ r(k i ) = ∞.
Therefore, we have which leads to contradictory, and completes the proof of necessity.
On the other hand, if r(t) is upper bounded, since w(t) − w(t − 1) is also upper bounded, we have 1 2 r(t) 2 + β 1−β r(t), w(t) − w(t − 1) is upper bounded, which completes the proof of sufficiency.
The proof is completed. Therefore, the last piece of this puzzle is to prove g(t) is upper bounded ∀t > 0.

Lemma 16 (Second argument in Lemma 2). Let all conditions in Theorem 2 hold. Then, we have g(t) is upper bounded.
Proof. We start the proof by calculating g(t + 1) − g(t). For any t ≥ 2, we have On the other hand, by simply rearranging the update rule Eq. (5), we have which further indicates Denote A 2 (t) = r(t + 1) − r(t) 2 , and A 3 (t) = r(t), − log t+1 t ŵ − η 1−β ∇L(w(t)) . We then prove respectively ∞ t=1 A 2 (t) and where Eq. (•) is due to Lemma 3 and log t+1 t = O( 1 t ). Then we only need to prove ∞ t=1 A 3 (t) < ∞. To begin with, by adding one additional term 1 tŵ into A 3 , we have On the one hand, by Corollary 3, w(t) = O( √ t), which further leads to On the other hand, by direct calculation of the gradient, we have We then analysis these two terms respectively. As for A 4 (t), due to ℓ ′ < 0, we have By Corollary 4, we further have ∀t > t 0 For each term r(t), − η we divide the analysis into two parts depending on the sign of r(t), x i .
We are now ready to prove Theorem 2.
Proof of Theorem 2. By Lemma 16, we have g(t) is upper bounded. Therefore, by Lemma 2, we have r(t) is bounded, which further indicates w(t) − log(t)ŵ is bounded. Therefore, the direction of w(t) can be calculated as The proof is completed.

B.2 IMPLICIT BIAS OF SGD WITH MOMENTUM
This section collects the proof of Theorem 3. Following the same framework as Appendix B.1, we will first prove that the sum of the squared gradient norms along the trajectory is finite. One may expect L(w(t)) + β 2η w(t) − w(t − 1) 2 is a Lyapunov function of SGDM. However, due to the randomness of the update rule of SGDM, L(w(t)) + β 2η w(t) − w(t − 1) 2 may no longer decrease (we will show this in the end of Appendix B.2, please see Appendix B.2.3 for explanation).

B.2.1 PROOF OF THE SUM OF GRADIENTS ALONG THE TRAJECTORY IS FINITE
We first provide a proof of Lemma 3.

Proof of Lemma 3.
We denote the parameter of the t-th step in Eq. (9) asw, while that in Eq. (1) as w(t). With w(1) =w(1), we will provew(t) = w(t), ∀t > 1 iteratively. Specifically, suppose for any 1 ≤ k ≤ t w(k) =w(k). Then, we have Therefore, which by iteration further indicates On the other hand, by Eq. (1), we have The proof is completed.
Using the alternative form given by Lemma 3, we then prove Lemma 4, which indicates L(u(t)) is a proper choice of Lyapunov function.
As L(u(1)) is upper bounded, we have the following corollary given by Lemma 4.
Proof. By Lemma 4, we have for any T > 1, which completes the proof of Eq. (38). The rest of claims follows immediately by Fubini's Theorem and Assumption 2.
The proof is completed.

B.2.2 BOUNDING THE ORTHOGONAL PART
Similar to the case of GDM, we definew as the solution of Eq. (12) with c = η (1−β)N . We also let n(t) be given by Lemma 9, and define r(t) in this case as Asw is a constant vector, and n(t) → 0 as t → ∞, we have w(t) − log(t)ŵ has bounded norm if and only if r(t) is upper bounded. Similar to the GDM case, we have the following equivalent condition of that r(t) is bounded.

Lemma 17. Let all conditions in Theorem 3 hold. Then, r(t) is bounded almost surely if and only if function g(t)
is upper bounded almost surely, where g : Z + → R is defined as Proof. To begin with, we prove that almost surely | t τ =2 r(τ ) − r(τ − 1), w(τ ) − w(τ − 1) | is upper bounded for any t. By Corollary 5, we have almost surly ∞ t=1 ∇L(w(t)) 2 < ∞.
On the other hand, for any w, we have Therefore, we have almost surely, which further leads to almost surely By the definition of r(t) (Eq. (39)), we further have where Inequality (⋆) is due to n(τ ) − n(τ − 1) = O( 1 τ ) and log τ +1 τ = O( 1 τ ). Therefore, g(t) is upper bounded almost surely is equivalent to 1 2 r(t) 2 + β 1−β r(t), w(t) − w(t − 1) is upper bounded, which can be shown to be equivalent with r(t) is bounded following the same routine as Lemma 2.
The proof is completed.
As the case of GDM, we only need to prove g(t) is upper bounded to complete the proof of Theorem 3.

Lemma 18. Let all conditions in Theorem 3 hold. We have g(t) is upper bounded.
Proof. Following the same routine as Lemma 2, we have where ∞ t=1 r(t + 1) − r(t) 2 is upper bounded.
On the other hand, by the definition of r(t) (Eq. (39)), we have while by Lemma 9, Combining the above two equations, we further have Therefore, we only need to prove By directly applying the form of ∇L(w(t)), we have We will investigate these two terms respectively. As w(t),x → ∞, ∀x ∈ T (S), a.s., we have a.s., there exists a large enough time t 0 , s.t., ∀t ≥ t 0 , ∀x ∈ T (S), x, w(t) > 0. Therefore, where Eq. (⋆) is due the definition of θ (Eq. (31)), and Eq. (•) is due to lim t→∞ e − w+n(t), On the other hand, A 6 (t) can be rewritten as If r(t),x i ≥ 0, we have for ε < 0.5, where Eq. (•) is due to n(t) = O( 1 t 0.5+ε ), and Eq. (⋄) is due to e − r(t),xi r(t),x i ≤ 1 e .
On the other hand, if r(t),x i < 0, we have If −2 > r(t),x i , then for large enough time t, e − r(t)+n(t),xi ≥ e 3 2 , 1 − e −µ− w(t),xi ≥ e − 1 2 , and 1 t e − w, Conclusively, if r(t),x i < 0, for large enough t, we have which further indicates, for large enough t, we have <∞.
The proof is completed.

B.2.3 EXPLANATION FOR PROPER LYAPUNOV FUNCTION
Based on the success of applying Lyapunov function L(w(t)) + β 2η w(t) − w(t − 1) 2 to analyze gradient descent with momentum, it is natural to try to extend this routine to analyze stochastic gradient descent with momentum. However, in this section, we will show such Lyapunov function is not proper to analyze SGDM as this will put constraints on the range of the momentum rate β. Specifically, at any step t, since the loss L is H smooth at w(t), we can expand the loss L in the same way as the GDM case: L(w(t + 1)) ≤L(w(t)) + w(t + 1) − w(t), ∇L(w(t)) + H 2 w(t + 1) − w(t) 2 .

C IMPLICIT BIAS OF ADAM
This section collects the proof of the convergent direction of Adam, i.e., Theorem 4. The methodology of this section bears great similarity with GDM, although the preconditioner of Adam requires specific treatment for analysis. The proof is still divided into two stages: (1). we first prove the sum of squared gradients along the trajectory is finite. Additionally, we prove the convergent rate of loss is O( 1 t ); (2). we prove w(t) − log(t)ŵ has bounded norm. Before we present these two stages of proof, we will first give the required range of η for which Theorem 3 holds.
The "sufficiently small learning rate" in Theorem 3 means .
To ensure η is well-defined, we need to prove and we introduce the following technical lemma: Proof. First of all, by definition, .
The left side of the above inequality is no smaller than 1, while the right side is no larger than 1, which completes the proof.
We are now ready to prove η is well-defined. First of all, for every t, we have where Eq. (⋆) is by Lemma 19 and cβ 1 = 4 √ β 2 < 1.
On the other hand, we have By Eq. (45) and Eq. (46), we obtain 1−(cβ1) t 1−(cβ1) t−1 is lower bounded by some positive constant across t, and η is well defined.

C.2 SUM OF GRADIENTS ALONG THE TRAJECTORY IS BOUNDED
We start with the following lemma, which indicates L(w(t))+ 4 ε1 d +ν(t)⊙(w(t)−w(t−1)) 2 is a proper Lyapunov function for Adam.

Preprint and under review
Eq. (47) contradicts the fact that the equality in Eq. (10) holds for 1 + α * , which completes the proof of t = 1.
This contradicts to that the equality in Eq. (10) holds for t * + α * .
The proof is completed.
and we only need to prove r(t) is bounded over time. We then prove Lemma 7, providing an equivalent condition of r(t) being bounded. As the GDM and SGDM case, we separate the proof into two sub-lemmas.
The proof is completed.

D.2 EXTENSION TO THE MULTI-CLASS CLASSIFICATION PROBLEM
Here we use several notations and lemmas from (Soudry et al., 2018). We define w = vec(W ), w = vec(Ŵ ), e i ∈ R C (i ∈ {1, · · · , C}) satisfying (e i ) j = δ ij , and A i = e i ⊗ I dX , where I dX is the identity matrix with dimension d X . We still consider the normalized data, i.e., x ≤ 1, ∀(x, y) ∈ S. Then, the individual loss of sample (x, y) can be then represented as ℓ(y, W x) = log e w,Ayx one can then easily verify all absolute value of the eigenvalues of HL(w) is no larger than 2, which indicates L is 2-globally smooth.
On the other hand, the separable assumption leads to ŵ, (A y − A i )x > 0, ∀y = i, which further indicates ∇L(w),ŵ > 0.
Let γ = 1 ŵ , following the similar routine as the binary case, we have for a random subset of S sampled uniformly without replacement with size b, we have Similarly, we have for any positive real series {a t } t2 t=t1 , γ t2 t=t1 a(t) ∇L(w(t)) ≤ t2 t=t1 a(t)∇L(w(t)) ≤ t2 t=t1 a(t) ∇L(w(t)) .
The proofs of Stage I can then be obtained with Lyapunov functions unchanged and by replacing the corresponding lemmas using Eq. (66) and Eq. (67).
The proof of Theorem 5 is then completed.