A termination criterion for stochastic gradient descent for binary classification

Abstract: We propose a new, simple, and computationally inexpensive termination test for constant step-size stochastic gradient descent (SGD) applied to binary classification on the logistic and hinge loss with homogeneous linear predictors. Our theoretical results support the effectiveness of our stopping criterion when the data is Gaussian distributed. This presence of noise allows for the possibility of non-separable data. We show that our test terminates in a finite number of iterations and when the noise in the data is not too large, the expected classifier at termination nearly minimizes the probability of misclassification. Finally, numerical experiments indicate for both real and synthetic data sets that our termination test exhibits a good degree of predictability on accuracy and running time.

\section{Analysis of stopping criterion} \label{sec:Analysis}
In this section, we present our analysis of the stopping criterion \eqref{eq: termination_test} proposed in Section~\ref{sec:termtest}. Here we introduce the first iteration at which the stopping criterion is satisfied, denoted by the random variable
 \begin{equation}\label{eq:stopping_criteria}
    T:=\inf\left\{k> 0: \hat{\bm{\xi}}_k^T\bm{\theta}_k \geq 1\right\}.
\end{equation}
By viewing the stopping criterion through the lens of stopping times, we are able to utilize probability theory to analyze the classifier at termination $\bm{\theta}_T$.  Throughout this section, we work with the following filtration.
\begin{equation}\label{eq:sigma}
  \mathcal{F}_0 = \sigma(\bm{\theta}_0) \quad \text{and} \quad  \mathcal{F}_k:=\sigma(\bm{\theta}_0, \hat{\bm{\xi}_1}, \bm{\xi}_1, \hat{\bm{\xi}_2}, \bm{\xi}_2, \hdots, \hat{\bm{\xi}_k}, \bm{\xi}_k), \quad \text{for all $k\geq 1$}
\end{equation}
 Clearly, the random variable $\bm{\theta}_k$ is $\mathcal{F}_k$-measurable.  Our theoretical results are structured as follows.

 First, we show that SGD with our proposed termination test indeed stops after a finite number of iterations. To do so, we provide a bound on $\bE[T]$, \textit{i.e.} the expected number of iterations before termination. Yet, despite this guarantee, the resulting classifier at termination need not be optimal. Hence, our second result establishes that both $\bm{\theta}_T$ and $\bm{\theta}^*$ point in approximately the same direction; thereby ensuring that the classifier at termination, $\bm{\theta}_T$, is nearly optimal. We remark the worst-case bounds established throughout these sections are conservative; we observe in our experiments that the termination test stops sooner while also yielding good classification properties for Gaussian and non-Gaussian data sets.

To bound $\bE[T]$, we identify subsets of $\R^d$ for which when an iterate enters the set, termination (\textit{i.e.} \eqref{eq: termination_test}) is \textit{highly likely} to succeed. Such sets $C$, we call \textit{target sets}. Precisely, for any $\bm{\theta} \in C$ and $\hat{\bm{\xi}}\sim N(\bm{\mu},\sigma^2I_d)$, the probability of terminating is at least $\delta>0$,
\begin{equation}\label{eq:probability_delta}
    \exists \; \delta>0 \text{ such that } \mathbb{P}_{\hat{\bm{\xi}}}\left( \hat{\bm{\xi}}^T\bm{\theta}\geq 1 \right) \geq \delta.
\end{equation}
We guarantee the iterates generated by SGD enter the target set by way of a \textit{drift function}, $V:\R^d\rightarrow [0,+\infty)$. A drift function, on average, decreases each time the iterate fails to live in the target set. In other words, conditioned on the past iterates the following holds
\begin{equation}\label{eq:driftequation}
    (\bE[V(\bm{\theta}_k)|\mathcal{F}_{k-1}]-V(\bm{\theta}_{k-1})])1_{\{\bm{\theta}_{k-1}\not\in C\}} \leq -b1_{\{\bm{\theta}_{k-1}\not\in C\}}
\end{equation}
for the target set $C$ and some positive constant $b$. Loosely speaking, the iterates in expectation \textit{drift} towards the target set. Target sets and drift functions in the context of drift analysis are well-studied in stochastic processes, see Lemma \ref{lem:drift_from_meyn} below.

A natural choice for the target set is a neighborhood of the unique optimum solution of \eqref{optimization_problem}, $\bm{\theta}^*$, with the drift function $\Vert \bm{\theta}-\bm{\theta}^* \Vert^2$. Indeed, it is known the iterates of SGD converge to a neighborhood of $\bm{\theta}^*$ (\cite{Pflug}). However, an iterate may be nearly optimal well before it enters this neighborhood. In fact when $\sigma \ll \Vert \bm{\mu} \Vert$, we identify a target set where satisfying the stopping criterion occurs at least half the time and does not require the iterate to be near $\bm{\theta}^*$. We summarize below our target set and drift function.

\begin{enumerate}
    \item Under the assumption $\sigma \leq c \Vert \bm{\mu} \Vert$ for some numerical constant $c$, which we call the \textit{Low Variance Regime}, we define the target set to be
    \begin{equation}\label{eq:targetset_lownoise}
        C = \{\bm{\theta}: \bm{\mu}^T\bm{\theta}\geq 1 \},
    \end{equation}
    and the drift function by
    \begin{equation}\label{eq:driftfunction_lownoise}
        V(\bm{\theta})=\left(M-\bm{\mu}^T\bm{\theta}\right)^2,
    \end{equation}
    for some constant $M$, to be determined later.
    \item Under the assumption $c\Vert \bm{\mu} \Vert \leq \sigma$ where the constant $c$ is the same as in 1 above, which we call the \textit{High Variance Regime}, we define the target set to be
    \begin{equation}\label{eq:targetset_highnoise}
        C=\{\bm{\theta}: \vert \rho\sigma^2-1\vert <1 \text{ and } \sigma \Vert \tilde{\bm{\theta}}\Vert\leq c'\},
    \end{equation}
    for some numerical constant $c'$. Here, we orthogonally decompose $\bm{\theta}=\rho \bm{\mu}+\tilde{\bm{\theta}}$ with $\bm{\mu}^T\tilde{\bm{\theta}}=0$. We use the following drift function
    \begin{equation}\label{eq:driftfunction_highnoise}
     V(\bm{\theta})=\frac{1}{2\alpha}\Vert \bm{\theta}-\bm{\theta}^*\Vert^2.
    \end{equation}
\end{enumerate}
In Section \ref{sec:proof_low} (resp. Section \ref{sec:proof_high}) we show that the pairs $(C,V)$ defined in \eqref{eq:targetset_lownoise} and \eqref{eq:driftfunction_lownoise} (resp. \eqref{eq:targetset_highnoise} and \eqref{eq:driftfunction_highnoise}) satisfies the drift equation \eqref{eq:driftequation} for any step-size $\alpha$ (resp. for any sufficiently small step-size $\alpha$).

As mentioned above, the target set $C$ attracts the iterates generated by SGD. Each time an iterate enters $C$, the stopping criterion holds with probability at least $\delta>0$. Provided the iterates enters the set $C$ an infinite number of times, then after waiting a geometrically distributed many iterations, we expect the following condition to hold:
\begin{equation}\label{eq:termination_inside_C}
    \hat{\bm{\xi}}_k^T\bm{\theta}_k\geq 1 \text{ and } \bm{\theta}_k \in C.
\end{equation}
 The SGD algorithm does not know the value of $\bm{\theta}^*$; therefore at each iteration, it cannot check whether the condition \eqref{eq:termination_inside_C} occurs. Nevertheless, we are able to compute a bound on the average waiting time until \eqref{eq:termination_inside_C} holds and the first time \eqref{eq:termination_inside_C} holds is always an upper bound on $T$, our stopping criterion. This is summarized in Lemma \ref{lem:ETleqET_C}. Precisely, if we denote by
 \begin{equation}\label{eq:stoppingtime_TC}
     T_C:=\inf\{k>0:\hat{\bm{\xi}}_k^T\bm{\theta}_k\geq 1 \text{ and } \bm{\theta}_k \in C\},
 \end{equation}
 then $T\leq T_C$, thus yielding $\bE[T]\leq \bE[T_C]$. We bound $\bE[T_C]$ by way of stopping times $\tau_m$ defined as the $m^{th}$ time the iterates of SGD enters $C$. Formally for any sequence $\{\bm{\theta}_k\}_{k=0}^\infty$ generated by SGD starting at $\bm{\theta}_0 = \bm{0}$, we set
 \begin{equation}\label{eq:tau1}
     \tau_1:=\inf\{k>0:\bm{\theta}_k\in C\}
 \end{equation}
and inductively, for $m\geq 2$,
\begin{equation}\label{eq:taum}
    \tau_m:=\inf\{k>\tau_{m-1}:\bm{\theta}_k\in C\}.
\end{equation}
The following lemma formalizes the discussion above.
\begin{lemma}\label{lem:ETleqET_C}
Let $\{\bm{\theta}_k\}_{k=0}^\infty$ be a sequence generated by SGD such that $\bm{\theta}_0 = \bm{0}$ and suppose that $\bE[\tau_m]<+\infty$ for all $m\geq 1$. Then the following holds
\begin{equation}
    \bE[T]\leq \bE[T_C]\leq \sum_{m=1}^{\infty} \bE[\tau_m](1-\delta)^{m-1},
\end{equation}
where $\delta$ satisfies \eqref{eq:probability_delta}.
\end{lemma}
\begin{proof}
 We first show that \begin{equation} \label{eq:analysis_1} \bE\left[1_{\{T_C\geq \tau_m\}}\right]\leq (1-\delta)^{m-1}.\end{equation} Define the $\sigma$-algebra $\mathcal{F'}=\sigma(\bm{\theta}_0,\bm{\xi}_1,\bm{\xi}_2,\cdots)$. From the independence between $\sigma(\hat{\bm{\xi}}_k)$'s  and $\mathcal{F}'$ and also $\tau_i<+\infty$ a.s. for all $i\geq 1$, the following is obtained:
\begin{align*}
    \bE\left[1_{\{ T_C\geq \tau_m\}}|\mathcal{F}'\right]
    &= \bE\left[1_{\{ \hat{\bm{\xi}}_{\tau_1}^T\bm{\theta}_{\tau_1}<1\}}\cdots1_{\{ \hat{\bm{\xi}}_{\tau_{m-1}}^T\bm{\theta}_{\tau_{m-1}}<1\}}|\mathcal{F}'\right]\\&=\prod_{i=1}^{m-1}\bE\left[1_{\{ \hat{\bm{\xi}}_{\tau_i}^T\bm{\theta}_{\tau_i}<1\}}|\mathcal{F}'\right]\\&\leq (1-\delta)^{m-1}.
\end{align*}
By taking expectations, we conclude \eqref{eq:analysis_1} holds. Now since $\bE[1_{\{T_C=+\infty\}}]\leq \bE[1_{\{T_C\geq \tau_m\}}]$ for all $m\geq 1$, it follows from \eqref{eq:analysis_1}  that $T_C<\infty$ a.s. We next observe that
\begin{align*}
    \bE\left[T_C1_{\{ T_C= \tau_m\}}|\mathcal{F}'\right]&=\bE\left[\tau_m1_{\{ T_C= \tau_m\}}|\mathcal{F}'\right]\\
    &\le\tau_m\bE\left[1_{\{ \hat{\bm{\xi}}_{\tau_{1}}^T\bm{\theta}_{\tau_{1}}<1\}}\cdots1_{\{ \hat{\bm{\xi}}_{\tau_{m-1}}^T\bm{\theta}_{\tau_{m-1}}<1\}}|\mathcal{F}'\right]\\
    &=\tau_m\prod_{i=1}^{m-1}\bE\left[1_{\{ \hat{\bm{\xi}}_{\tau_{i}}^T\bm{\theta}_{\tau_{i}}<1\}}|\mathcal{F}'\right]\\
    &\leq \tau_m(1-\delta)^{m-1}.
\end{align*}
Taking expectations yields $\bE\left[T_C1_{\{T_C=\tau_m\}}\right]\leq \bE\left[\tau_m\right](1-\delta)^{m-1}$ for all $m\geq 1$. Now since $T_C<\infty$ a.s. we get $1=\sum_{m=1}^{+\infty}1_{\{T_C=\tau_m\}}$ a.s. This yields that
\[
    \bE[T]\leq \bE[T_C]=\sum_{m=1}^{\infty} \bE[T_C1_{\{T_C=\tau_m\}}]\leq \sum_{m=1}^{\infty}\bE[\tau_m](1-\delta)^{m-1}.
\]
The proof is complete.
\end{proof}
 Now, in view of Lemma \ref{lem:ETleqET_C}, it suffices to bound $\bE[\tau_m]$ by a sequence which can not grow too fast in $m$. Indeed, we show that \eqref{eq:driftequation} implies the following
 \begin{equation}
     \bE[\tau_m]=\mathcal{O}(m).
 \end{equation}
\begin{theorem}(Low Regime)\label{thm:low}
Let $\{\bm{\theta}_k\}_{k=0}^\infty$ be a sequence generated by Algorithm~\ref{alg:SGD_termination} such that $\bm{\theta}_0=\bm{0}$. There exists positive constants $c,b$ and $M$ such that provided $\sigma \leq c\Vert \bm{\mu} \Vert$ the following holds.
\begin{equation}
    \bE[T]\leq 2+\frac{2M^2}{b}\cdot\left(\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\frac{\alpha\sigma^3}{ \Vert \bm{\mu} \Vert }\cdot\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right)+1 \right).
\end{equation}
Here the constants $c,b$ and $M$ are defined as follows:
\begin{enumerate}
    \item For the logistic loss,
    \begin{equation}\label{eq:para_log}
    c=0.33,\quad b=\alpha\Vert \bm{\mu} \Vert^2,\quad \text{ and } M=501+640\alpha \Vert \bm{\mu} \Vert^2.
\end{equation}
\item For the hinge loss,
\begin{equation}\label{eq:para_hinge}
    c=1.25,\quad b=\alpha\Vert \bm{\mu} \Vert^2,\quad \text{ and } M=501+782\alpha \Vert \bm{\mu} \Vert^2.
\end{equation}
\end{enumerate}
\end{theorem}
Therefore, on relatively separable data (\textit{i.e.} in the low variance regime), the expected waiting time before termination exponentially decreases as the data becomes more separable (\textit{i.e.} $\sigma \to 0$).
We prove Theorem \ref{thm:low} in Section \ref{sec:theorem_angle}. The next theorem shows that the expected value of the stopping time is finite provided that the $\sigma > c\Vert \bm{\mu} \Vert$ and the step-size is small enough.
\begin{theorem}\label{thm:high}(High Regime) Suppose that $\sigma > c\Vert \bm{\mu} \Vert$ where $c$ is defined in \eqref{eq:para_log} and \eqref{eq:para_hinge}. Then there exists a universal positive constant $A$ such that if the step-size $\alpha$ satisfies
\begin{equation}
    \alpha \leq A\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2(\Vert \bm{\mu} \Vert^2+d\sigma^2)},
\end{equation}
then it holds that $\bE[T]<+\infty$. In particular, the termination criterion occurs almost surely.
\end{theorem}
It remains to determine whether the classifier at termination $\bm{\theta}_T$, has desirable accuracy. The scale-invariance of optimal classifiers means a classifier yields a lower probability of misclassification the closer its direction aligns with any optimal classifier. In view of this, it suffices to bound the absolute value of the inner product of any unit vector that is perpendicular to $\bm{\theta}^*$, $\bm{v}$ with $\bm{\theta}_T$. The following theorem establishes a bound on $\bE[\vert \bm{v}^T\bm{\theta}_T\vert]$.
\begin{theorem}\label{thm:angle_bound}
Let $\bm{\theta}_0=\bm{0}$. Fix any unit vector $\bm{v}\in \R^d$ such that $\bm{v}^T\bm{\theta}^*=0$. Then the following estimate holds
\begin{equation}
    \bE[\vert \bm{v}^T\bm{\theta}_T\vert]\leq \sigma \alpha \sqrt{\frac{2}{\pi}} \bE[T].
\end{equation}
\end{theorem}
In the low variance regime by combining Theorem \ref{thm:low} and \ref{thm:angle_bound} for a fixed step-size $\alpha$ it holds that $\bE[\vert \bm{v}^T\bm{\theta}\vert]\leq \mathcal{O}(\sigma)$. Thus, the more separable the data set is, the more accurate the classifier $\bm{\theta}_T$ is on average. In the high variance regime, Theorem \ref{thm:high} yields a very loose bound. Yet despite this, our numerical result in Section \ref{sec:Num_Experiment} show promising accuracy of \eqref{eq: termination_test} in this case as well. We conjecture that the inequality can be significantly strengthened.
\input{arxiv_low_variance_regime.tex}
\input{arxiv_high_variance_regime.tex}
\input{arxiv_angle_bound.tex}
\subsection{Angle bound, proof of Theorem \ref{thm:angle_bound}}\label{sec:theorem_angle}

\begin{proof}[Proof of Theorem \ref{thm:angle_bound}] Recall the SGD algorithm for logistic regression uses the update
\[
\bm{\theta}_k=\bm{\theta}_{k-1}+\frac{\alpha\bm{\xi}_k}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}
\]
and for hinge regression
\[
\bm{\theta}_k=\bm{\theta}_{k-1}+\alpha 1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\bm{\xi}_{k-1}
\]
where $\bm{\theta}_0=\bm{0}$ and $\bm{\xi}_1,\bm{\xi}_2,\cdots \stackrel{i.i.d}{\sim} N(\bm{\mu},\sigma^2I_d)$.  It clearly holds in both cases that
\begin{equation}
    \left\vert\vert \bm{v}^T\bm{\theta}_{k}\vert-\vert \bm{v}^T\bm{\theta}_{k-1}\vert \right\vert\leq \alpha \vert \bm{v}^T\bm{\xi}_{k-1}\vert.
\end{equation}
We define a new random variable $X_k:=\vert \bm{v}^T\bm{\theta}_k\vert-k \sigma\alpha \sqrt{\frac{2}{\pi}} $. Observe that $\bE\left[\left|X_0\right|\right] = 0$ and for all $k \ge 1$, it holds that
\begin{align*}
    \bE\left[\left\vert X_k\right\vert\right]&\leq \alpha\sum_{i=1}^k \bE\left[\left\vert \bm{v}^T\bm{\xi}_k \right\vert\right]+k\sigma\alpha \sqrt{\frac{2}{\pi}} <\infty,
\end{align*}
\textit{i.e.}, $X_k \in \mathcal{L}^1$ for all $k \geq 1$. Next, we have for any $k \ge 1$
\begin{align*}
  \bE\left[\left\vert X_{k}-X_{k-1}\right\vert\,|\,\mathcal{F}_{k-1} \right] \leq  \bE\left[\left\vert \vert \bm{v}^T\bm{\theta}_k\vert -\vert \bm{v}^T\bm{\theta}_{k-1}\vert\right\vert\,|\,\mathcal{F}_{k-1} \right]+\sigma\alpha  \sqrt{\frac{2}{\pi}}  &\leq 2\sigma\alpha  \sqrt{\frac{2}{\pi}} .
\end{align*}
Here we used that $\bm{v}^T\bm{\xi}_k \sim N(0,\sigma^2)$ along with \eqref{fact:norm_Gaussians}. We also see that \begin{align*}
    \bE\left[\vert \bm{v}^T\bm{\theta}_{k}\vert\,|\,\mathcal{F}_{k-1} \right]&\leq \vert \bm{v}^T\bm{\theta}_{k-1}\vert+ \sigma \alpha\sqrt{\frac{2}{\pi}} \quad \Rightarrow \quad  \bE\left[X_{k}\,|\,\mathcal{F}_{k-1} \right]\leq X_{k-1}.
\end{align*}
Therefore, we have shown that $X_0,X_1,\cdots$ is a super-martingale. By Theorem \ref{thm:Durret_Martingale}, we have $\bE\left[X_{T}\right]\leq 0$. The result follows.

\end{proof}

\section{Appendix}
In this section, we provide proofs of technical results that we used in the paper.
\subsection{Drift Analysis Lemma}
Here we state and prove a result on Drift Analysis that we used in the paper. Recall that we require the following equation to hold
\begin{equation}\label{drift_equation1_appendix}
  \left( \bE\left[V(\bm{\theta}_k)|\mathcal{F}_{k-1} \right]-V(\bm{\theta}_{k-1}) \right)1_{\{\bm{\theta}_{k-1}\not\in C\}}\leq -1_{\{\bm{\theta}_{k-1}\not\in C\}}.
\end{equation}
However, when the iterate lies inside $C$, then we do not assume any bound on the expected increase in the drift function. Nevertheless, when the target set $C$ is compact then there exists a positive constant $b>0$ such that
\begin{equation}\label{drift_equation2_appendix}
  \left( \bE\left[V(\bm{\theta}_k)|\mathcal{F}_{k-1} \right]-V(\bm{\theta}_{k-1}) \right)1_{\{\bm{\theta}_{k-1}\in C\}}\leq b1_{\{\bm{\theta}_{k-1}\in C\}},
\end{equation}
The following lemma bounds the expected value of return times $\tau_m^C$.
\begin{lemma}\label{lem:drift_lemma_appendix} Suppose that for some test function $V$ and a target set $C$, the drift equation \eqref{drift_equation1_appendix} holds. Let $\hat{\bm{\theta}} \in \R^d\backslash C$. The following is then true
\begin{equation}
    \bE[\tau_1^C|\bm{\theta}_0=\hat{\bm{\theta}}]\leq V(\hat{\bm{\theta}}).
\end{equation}
In addition, if both equations \eqref{drift_equation1_appendix} and \eqref{drift_equation2_appendix} hold, then the following is true
\begin{equation}
    \bE[\tau_m^C|\bm{\theta}_0=\hat{\bm{\theta}}]\leq V(\hat{\bm{\theta}})+(m-1)\sup_{\bm{\theta}\in C} V(\bm{\theta}).
\end{equation}
\end{lemma}
\begin{proof}
 See \cite{meyn2012markov}.
\end{proof}
\subsection{Stopping Time Lemma}\label{stopping_time_lemma}
In light of drift equation \eqref{drift_equation_appendix}, the iterates are inclined towards the target set $C$. With a positive probability $\delta$, the test will be activated once we are in $C$. Hence, the expected value of the stopping time $T$, i.e. $\bE[T]$, can be upper-bounded in terms of $\delta$ and expected return times $\bE[\tau_m^C]$.
\begin{lemma}\label{lem:stopping_time_appendix}
Suppose that $\bE[\tau_m^{C}]<\infty$ for all $m\geq 1$.  Then it is true that $T_C<+\infty$. Moreover, the following holds.
\begin{equation}\label{eq:ST_tctm}
   \bE[T]\leq \bE[T_C]\leq \sum_{m=1}^{\infty}\bE[\tau_m^{C}](1-\delta)^{m-1},
\end{equation}
where for any fixed $\bm{\theta}\in C$ the constant $\delta$ satisfies
\begin{equation}
    \delta= \min \mathbb{P}_{\bm{\xi} \sim \mathcal{P}_*}\left(\bm{\xi}^T\bm{\theta}\geq 1 \right).
\end{equation}
\end{lemma}

\subsection{Basic Convex Analysis Lemma}
Here we state and prove two basic results from convex analysis that we used in the paper.
\begin{lemma}\label{lem:convex_analysis_lemma}
Suppose that $g:\R^{\geq 0}\to \R$ is a convex function with a minimizer at $\rho^*>0$. Assume that $g$ is twice differentiable on the interval $[\tfrac{3}{4}\rho^*,\tfrac{5}{4}\rho^*]$. Moreover, assume that there exists a constant $r>0$ such that $g''(\rho)\geq r$ for all $\rho \in [\tfrac{3}{4}\rho^*,\tfrac{5}{4}\rho^*]$. Then it holds that
\begin{equation}\label{eq0:cvx_lemma}
    g(\rho)-g(\rho^*)\geq \frac{\rho^*r}{8}\vert \rho-\rho^*\vert \quad \text{for all $\rho \not\in [\tfrac{1}{2}\rho^*,\tfrac{3}{2}\rho^*]$}.
\end{equation}
\end{lemma}
\begin{proof}
First, assume that $\rho > \tfrac{3}{2}\rho^*$ holds. There exists  $\hat{\rho}\in [\rho^*,\tfrac{5\rho^*}{4}]$ such that
\begin{equation}\label{eq1:cvx_lemma}
    g'(\rho^*+\frac{\rho^*}{4})=g'(\rho^*+\frac{\rho^*}{4})-g'(\rho^*)=\frac{\rho^*}{4}g''(\hat{\rho})\geq \frac{\rho^*r}{4},
\end{equation}
where we used that $g'(\rho^*)=0$. With the convexity of $g$, for any $\rho>\frac{3\rho^*}{2}$, we have
\[
g(\rho)\geq g(\rho^*+\frac{\rho^*}{4})+g'(\rho^*+\frac{\rho^*}{4})(\rho-\frac{5\rho^*}{4})\geq g(\rho^*)+\frac{\rho^*r}{4}(\rho-\frac{5\rho^*}{4}).
\]
Note that the second inequality follows from \eqref{eq1:cvx_lemma} and the optimal value of $g$ occurring at $g(\rho^*)$. Using the identity, $2(\rho-\frac{5\rho^*}{4})\geq \frac{\rho^*}{4}+\rho-\frac{5\rho^*}{4}=\rho-\rho^*$, we conclude
\begin{equation}\label{eq:cvx_lemma1}
    g(\rho)-g(\rho^*)\geq \tfrac{1}{8}\rho^*r(\rho-\rho^*) \quad \text{for all $\rho > \frac{3\rho^*}{2}$}.
\end{equation}
We now consider the second case, $\rho<\frac{\rho^*}{2}$. Similarly as above, we get
\begin{equation}\label{eq2:cvx_lemma}
    -g'(\rho^*-\frac{\rho^*}{4})= g'(\rho^*)-g'(\rho^*-\frac{\rho^*}{4})=\frac{\rho^*}{4}g''(\hat{\rho})\geq \frac{\rho^*r}{4}.
    \end{equation}
    for some $\hat{\rho}\in [\frac{3\rho^*}{4},\rho^*]$. Since $\rho<\frac{\rho^*}{2}$, we have
    \[
    g(\rho)\geq g(\rho^*-\frac{\rho^*}{4})+g'(\rho^*-\frac{\rho^*}{4})(\rho-\frac{3\rho^*}{4})\geq g(\rho^*)+\frac{\rho^*r}{4}(\frac{3\rho^*}{4}-\rho).
    \]
    Here, again, the first inequality follows from convexity of $g$ and the second inequality from \eqref{eq2:cvx_lemma} and $g(\rho^*-\frac{\rho^*}{4})\geq g(\rho^*)$. We note that $2(\frac{3\rho^*}{4}-\rho)\geq \frac{\rho^*}{4}+\frac{3\rho^*}{4}-\rho=\rho^*-\rho$ and therefore
    \begin{equation}\label{eq:cvx_lemma2}
        g(\rho)-g(\rho^*) \geq \frac{\rho^*r}{8}(\rho^*-\rho)\quad \text{for all $\rho < \frac{\rho^*}{2}$}.
    \end{equation}
    Combining \eqref{eq:cvx_lemma1} and \eqref{eq:cvx_lemma2} the result follows.
\end{proof}
We used the following technical lemma in the paper as well.
\begin{lemma}\label{lem:technical_convex_bound}
Consider the following optimization problem
\[
\min f(\bm{\theta}):=\bE_{(\bm{\xi},y)\sim \mathcal{P}_*}\left[\ell(\bm{\xi}^T\bm{\theta})\right].
\]
where $\ell$ is a convex function. The sequences $\{\bm{\theta}_k, \bm{\xi}_k \}_{k = 0}^\infty$ generated by Algorithm~\ref{alg:SGD_termination} satisfy for any $\bm{\theta} \in \R^d$, the following
    \begin{equation} \label{eq:high_decrease}
       f(\bm{\theta}_{k-1})-f(\bm{\theta})\leq \frac{1}{2\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta} \Vert^2-\bE\left[\Vert \bm{\theta}_{k}-\bm{\theta} \Vert^2\, |\mathcal{F}_{k-1}\right] \right)+\bE[\Vert\nabla_{\bm{\theta}}\ell\left( \bm{\xi}_k^T\bm{\theta}\right)\Vert^2 \, | \, \mathcal{F}_{k-1}],
    \end{equation}
    for all $k\geq 1$ where $f$ is defined in \eqref{optimization_problem} and the filtration $\{\mathcal{F}_k\}_{k=0}^{+\infty}$ in \eqref{eq:sigma}.
\end{lemma}
\begin{proof
 Define the quantity
  \begin{equation*}
      \bm{g_k}:=\frac{1}{\alpha}\left( \bm{\theta_{k-1}}-\bm{\theta_{k}}\right) = \nabla_{\bm{\theta}}\ell\left(\bm{\xi}_k^T\bm{\theta} \right). \end{equation*}
      Therefore, we obtain that
      \[
      \bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right]=\nabla_{\bm{\theta}} f(\bm{\theta}_{k-1})
      \]
By convexity of the function $f$, we have for any $\bm{\theta} \in \R^d$ the following
\begin{align*}
\norm{\bm{\theta}_{k}-\bm{\theta}}^2 &= \norm{\bm{\theta}_{k-1}-\bm{\theta}}^2 -2 \alpha \bm{g}_k^T (\bm{\theta}_{k-1}-\bm{\theta}) + \alpha^2\norm{\bm{g}_k}^2\\
&= \norm{\bm{\theta}_{k-1}-\bm{\theta}}^2 - 2\alpha(\bm{g}_k-\bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right])^T(\bm{\theta}_{k-1}-\bm{\theta}) - 2 \alpha \bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right]^T (\bm{\theta}_{k-1}-\bm{\theta})+\alpha^2 \norm{\bm{g}_k}^2\\
&\le \norm{\bm{\theta}_{k-1}-\bm{\theta}}^2- 2\alpha(\bm{g}_k-\bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right])^T(\bm{\theta}_{k-1}-\bm{\theta})-2\alpha (f(\bm{\theta}_{k-1})-f(\bm{\theta})) + \alpha^2 \norm{\bm{g}_k}^2.
  \end{align*}
By taking conditional expectations with respect to $\mathcal{F}_{k-1}$ and rearranging the above inequality, the result follows.
\end{proof}

\section{Background and preliminaries}\label{sec:defs}
 Throughout we consider a Euclidean space, denoted by $\R^d$, with an inner product and an induced norm $\norm{\cdot}$. The set of non-negative real numbers is denoted by $\R_{\geq 0}$. Bold-faced variables are vectors. Throughout, the matrix $I_d$ is the $d$ by $d$ identity matrix. All stochastic quantities defined hereafter live on a probability space denoted by $(\bP, \Omega, \mathcal{F})$, with probability measure $\bP$ and the $\sigma$-algebra $\mathcal{F}$ containing subsets of $\Omega$. Recall, a random variable (vector) is a measurable map from $\Omega$ to $\R$ ($\R^d$), respectively. An important example of a random variable is the \textit{indicator of the event $A \in \mathcal{F}$}:
\[1_A(\omega) = \begin{cases}
1, & \omega \in A\\
0, & \omega \not \in A.
\end{cases}\]
If $X$ is a measurable function and $t \in \R$, we often simplify the notation for the pull back of the function $X$, to simply $\{\omega \in \Omega \, :\, X(\omega) \le t\} =: \{X \le t\}$.
As is often in probability theory, we will not explicitly define the space $\Omega$, but implicitly define it through random variables. For any sequence of random vectors $(\bm{X}_1, \bm{X}_2, \hdots, \bm{X}_k)$, we denote the \textit{$\sigma$-algebra generated by random vectors $\bm{X}_1, \bm{X}_2, \hdots, \bm{X}_k$} by the notation $\sigma(\bm{X}_1, \bm{X}_2, \bm{X}_3, \hdots, \bm{X}_k)$ and the \textit{expected value of $\bm{X}$} by $\bE[\bm{X}] := \int_{\Omega} \bm{X} \, d\bP$.

Particularly, we are interested in random variables that are distributed from normal distributions. In the next section, we state some known results about normal distributions.

\paragraph{Normal distributions}
The \textit{probability density function of a univariate Gaussian} with mean $\mu$ and variance $\sigma^2$ is described by:
\begin{equation*}
    \varphi(t) := \frac{1}{\sigma\sqrt{2\pi }}\exp\left(-\frac{(t-\mu)^2}{\sigma^2}\right).
\end{equation*}
In particular, we say a random variable $\xi$ is distributed as a Gaussian with mean $\mu$ and variance $\sigma^2$ by $\xi \sim N(\mu,\sigma^2)$ to mean $\bP(\xi \le t) = \int_{-\infty}^t \varphi(t) \, dt$. When the random variable $\xi \sim N(0,1)$, we denote its cumulative density function as
\[\Phi(t) := \bP(\xi \le t) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^t \exp\left (-\xi^2 \right )  \, d\xi,\]
and its complement by $\Phi^c(t) = 1-\Phi(t)$. The symmetry of a normal around its mean yields the identity, $\Phi(t) = \Phi^c(-t)$.

One can, analogously, formulate a higher dimensional version of the univariate normal distribution called a \textit{multivariate normal distribution}. A random vector is a multivariate normal distribution if every linear combination of its component is a univariate normal distribution. We denote such multivariate normals by $\bm{\xi} \sim N(\bm{\mu},\Sigma)$ with $\bm{\mu}\in \R^d$ and $\Sigma$ is a symmetric positive semidefinite $d\times d$ matrix.

Normal distributions have interesting properties which simplify our computations throughout the paper. We list those which we specifically rely on.  See \cite{Famoye_univardist_book} for proofs. Below, $\bm{v},\bm{v}'\in \R^d$, $r\in \R$, $\bm{\xi} \sim N(\bm{\mu},\sigma^2I_d)$  and $\xi\sim N(\mu,\sigma^2)$. Also, $\psi \sim N(0,1)$.

Throughout our analysis, we encounter random variables of the form $\bm{v}^T\bm{\xi}+r$, i.e. affine transformations of a given normal distribution. A fundamental property of normal distributions is that they stay in the same class of distributions after any such transformation. In other words, it holds that
\begin{equation}\label{eq:fact_affine}
    \bm{v}^T\bm{\xi}+r\sim N(\bm{v}^T\bm{\mu}+r, \sigma^2\Vert \bm{v} \Vert^2).
\end{equation}
Working with independent random variables makes the analysis significantly easier.  In particular, it is essential for us to know when the two random variables $\bm{v}^T\bm{\xi}$ and $\bm{v}'^T\bm{\xi}$ are independent. We will use the following simple fact below: The following is true
\begin{equation} \label{eqn:fact_independence}
\mbox{$\bm{v}^T\bm{\xi}$ and  $\bm{v}'^T\bm{\xi}$ are independent} \quad \text{ if and only if } \quad \bm{v}^T\bm{v}'=0.
\end{equation}
We will also use the following simple fact about truncated normal distributions:
\begin{equation} \label{eqn:fact_truncated}
\bE_{\xi}[\xi1_{\{\xi\leq b\}}]=0  \Longrightarrow \Phi\left(\frac{b-\mu}{\sigma}\right)\cdot \exp\left(\frac{1}{2}\cdot\left(\frac{b-\mu}{\sigma}\right)^2 \right)=\frac{\sigma}{\mu}.
\end{equation}
We conclude our remarks on normal distributions with the statement of two facts about the expected value of their norm. The following hold:
\begin{equation}\label{fact:norm_Gaussians}
    \bE\left[\Vert \bm{\xi} \Vert^2 \right]= \Vert \bm{\mu} \Vert^2+d\sigma^2, \quad \bE_{\xi}[\vert \xi \vert]\leq \sqrt{\frac{2}{\pi}}\cdot \sigma+\vert\mu\vert \quad \text{and}  \quad \bE\left[\vert \psi \vert \right] = \sqrt{\frac{2}{\pi}}.
\end{equation}

\paragraph{Martingales and stopping times} Here we state some relevant definitions and theorems used in analyzing our stopping criteria in Section~\ref{sec:Analysis}. We refer the reader to \cite{Durrett_probability_book} for further details. For any probability space, $\left(\bP,\Omega,\mathcal{F}\right)$, we call a sequence of $\sigma$-algebras, $\{\mathcal{F}_k\}_{k=0}^\infty$, a \textit{filtration} provided that $\mathcal{F}_i \subset \mathcal{F}$ and $\mathcal{F}_0 \subseteq \mathcal{F}_1 \subseteq \mathcal{F}_2 \subseteq \cdots$ holds. Given a filtration, it is natural to define a sequence of random variables $\{X_k\}_{k=0}^\infty$ with respect to the filtration, namely $X_k$ is a $\mathcal{F}_k$-measurable function. If, in addition, the sequence satisfies
\[\bE[|X_k|] < \infty \quad \text{and} \quad \bE[X_{k+1} | \mathcal{F}_k] \le X_k \quad \text{for all $k$},\]
we say $\{X_k\}_{k=0}^\infty$ is a  \textit{supermartingale}. In probability theory, we are often interested in the (random) time at which a given stochastic sequence exhibits a particular behavior. Such random variables are known as \textit{stopping times}. Precisely, a stopping time is a random variable $T: \Omega \to \mathbb{N} \cup \{0, \infty\}$ where the event $\{T=k\} \in \mathcal{F}_k$ for each $k$, i.e., the decision to stop at time $k$ must be measurable with respect to the information known at that time. Supermartingales and stopping times are closely tied together, as seen in the theorem below, which gives a bound on the expectation of a stopped supermartingale.
\begin{theorem}[See \cite{Durrett_probability_book} Theorem 4.8.5]\label{thm:Durret_Martingale}
Suppose that $\left\{X_k\right\}_{k=0}^{\infty}$ is a supermartingale w.r.t to the filtration $\left\{\mathcal{F}_k\right\}_{k=0}^{\infty}$ and let $T$ be any stopping time satisfying $\bE[T]<\infty$. Moreover if $\bE\left[\vert X_{k+1}-X_k\vert|\mathcal{F}_k\right]\leq B$ a.s. for some constant $B>0$, then it holds that $\bE[X_T]\leq \bE[X_0]$.
\end{theorem}
As we illustrate in Section~\ref{sec:Analysis}, a connection between stopping criteria (i.e. the decision to stop an algorithm) and stopping times naturally exists.
\subsection{High regime, proof of Theorem \ref{thm:high}}\label{sec:proof_high}
In this section, we consider the high variance regime. We consider the target set $C$ and the function $V$ defined in \eqref{eq:targetset_highnoise} and \eqref{eq:driftfunction_highnoise}, respectively, \textit{i.e.}
\begin{equation}\label{eq:CV_high_recall}
    C:=\left\{\bm{\theta}: \vert \rho -\rho^*\vert<\tfrac{1}{2}\rho^* \text{ and } \sigma \Vert \tilde{\bm{\theta}}\Vert \leq c'\right\} \quad \text{and}\quad V(\bm{\theta}):=\frac{1}{2\alpha}\Vert \bm{\theta}-\bm{\theta}^*\Vert^2,
\end{equation}
where the minimizer $\bm{\theta}^*=\rho^*\bm{\mu}$ is defined in Lemma \ref{lem:minimizers} and the constant $c'$ is to be determined. We first aim to show that $V$ is a drift function with respect to the set $C$ under the high variance regime assumption, meaning $\sigma \geq c\Vert \bm{\mu} \Vert$. We next state a standard SGD convergence result applied to the logistic and hinge loss functions.
\begin{lemma}\label{lem:technical_convex_bound}
Consider the optimization problem \eqref{optimization_problem} where $\ell:\R\times \R \rightarrow \R$ is either the logistic or hinge loss function. Denote the vector $\bm{\theta}^*$ as the unique minimizer of $f$ in \eqref{optimization_problem}. Let $\bm{\theta}_0\in \R^d$. The sequence $\{\bm{\theta}_k \}_{k = 0}^\infty$ generated by SGD satisfies the following for all $k\geq 1$,
    \begin{equation} \label{eq:high_convex_tech_lemma}
       f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)\leq \frac{1}{2\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta} ^*\Vert^2-\bE\left[\Vert \bm{\theta}_{k}-\bm{\theta}^* \Vert^2\, |\mathcal{F}_{k-1}\right] \right)+\frac{\alpha}{2}\left(\Vert \bm{\mu} \Vert^2+d\sigma^2\right).
    \end{equation}

\end{lemma}
\begin{proof
 Define the quantity
  \begin{equation*}
      \bm{g_k}:=\frac{1}{\alpha}\left( \bm{\theta}_{k-1}-\bm{\theta}_{k}\right) = \nabla_{\bm{\theta}}\ell\left(\bm{\xi}_k^T\bm{\theta}_{k-1},1 \right). \end{equation*}
Here, it is easy to check that the derivative with respect to $\bm{\theta}$ and the expectation over $\bm{\xi}_k$ are interchangeable, thus yielding      \[
      \bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right]=\nabla_{\bm{\theta}} f(\bm{\theta}_{k-1}).
      \]
By convexity of the function $f$, we have the following
\begin{align*}
\norm{\bm{\theta}_{k}-\bm{\theta}^*}^2 &= \norm{\bm{\theta}_{k-1}-\bm{\theta}^*}^2 -2 \alpha \bm{g}_k^T (\bm{\theta}_{k-1}-\bm{\theta}^*) + \alpha^2\norm{\bm{g}_k}^2\\
&= \norm{\bm{\theta}_{k-1}-\bm{\theta}^*}^2 - 2\alpha(\bm{g}_k-\bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right])^T(\bm{\theta}_{k-1}-\bm{\theta}^*) - 2 \alpha \bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right]^T (\bm{\theta}_{k-1}-\bm{\theta}^*)+\alpha^2 \norm{\bm{g}_k}^2\\
&\le \norm{\bm{\theta}_{k-1}-\bm{\theta}^*}^2- 2\alpha(\bm{g}_k-\bE_{\bm{\xi}_{k}}\left[\bm{g}_k|\mathcal{F}_{k-1} \right])^T(\bm{\theta}_{k-1}-\bm{\theta}^*)-2\alpha (f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)) + \alpha^2 \norm{\bm{g}_k}^2.
  \end{align*}
  By taking conditional expectations with respect to $\mathcal{F}_{k-1}$ and rearranging the above inequality, we obtain that
    \begin{equation}\label{eq:label_ineq_blah1}
       f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)\leq \frac{1}{2\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta} ^*\Vert^2-\bE\left[\Vert \bm{\theta}_{k}-\bm{\theta}^* \Vert^2\, |\mathcal{F}_{k-1}\right] \right)+\frac{\alpha}{2}\bE_{\bm{\xi}_k}\left[\Vert \nabla_{\bm{\theta}} \ell(\bm{\xi}_k^T\bm{\theta}_{k-1},1)\Vert^2 \right].
    \end{equation}
    We next observe the following bound
    \begin{equation}\label{eq:label_ineq_blah2}
        \bE_{\bm{\xi}_k}[\Vert\nabla_{\bm{\theta}}\ell\left( \bm{\xi}_k^T\bm{\theta}_{k-1},1\right)\Vert^2 \, | \, \mathcal{F}_{k-1}]\leq \bE_{\bm{\xi}_k}[\Vert \bm{\xi}_k \Vert^2|\mathcal{F}_{k-1}]=\Vert \bm{\mu} \Vert^2+d\sigma^2.
\end{equation}
Combining \eqref{eq:label_ineq_blah1} and \eqref{eq:label_ineq_blah2}, the result follows.
\end{proof}
By Lemma \ref{lem:technical_convex_bound} for each $k\geq 1$, we deduce
\begin{equation}\label{eq:driftequation_high}
\bE[V(\bm{\theta}_k)|\mathcal{F}_{k-1}]-V(\bm{\theta}_{k-1})\leq -\left(f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)\right)+\frac{\alpha}{2}(\Vert\bm{\mu}\Vert^2+d\sigma^2).
\end{equation}
Therefore, in order to show that the pair $(C,V)$ in \eqref{eq:CV_high_recall} satisfies the drift equation \eqref{eq:driftequation}, it suffices to lower bound the quantity $f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)$ whenever $\bm{\theta}_{k-1}\not\in C$. To do so, we orthogonally decompose $\bm{\theta}_{k-1}=\rho_{k-1}\bm{\mu}+\tilde{\bm{\theta}}_{k-1}$, \textit{i.e.} $\bm{\mu}^T\tilde{\bm{\theta}}_{k-1}=0$ and $\rho_{k-1}\in \R$ and write
\begin{equation}\label{eq:quantity_split}
\begin{aligned}
f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)= & \underbrace{f(\bm{\theta}_{k-1})-f(\rho_{k-1}\bm{\mu})}_{(a)}+\underbrace{f(\rho_{k-1}\bm{\mu})-f(\bm{\theta}^*)}_{(b)}.
\end{aligned}
\end{equation}
The assumption $\bm{\theta}_{k-1}\not \in C$ yields that either $\sigma \Vert \tilde{\bm{\theta}}_{k-1}\Vert \geq c'$ or $\vert \rho_{k-1}-\rho^*\vert\geq \tfrac{1}{2}\rho^*$. In Lemma \ref{lem:lower_bound_1} (resp. \ref{lem:lower_bound_2}), we show that (a) (resp. (b)) in \eqref{eq:quantity_split} are both non-negative and they are lower bounded by some positive constant provided that $\sigma \Vert \tilde{\bm{\theta}}_{k-1}\Vert \geq c'$ and $\vert \rho_{k-1}-\rho^*\vert\leq \tfrac{1}{2}\rho^*$ (resp. $\vert \rho_{k-1}-\rho^*\vert\geq \tfrac{1}{2}\rho^*$).
\begin{lemma}(Lower bound for (a) in \eqref{eq:quantity_split})\label{lem:lower_bound_1}
Fix $\bm{\theta}\in \R^d$ and orthogonally decompose $\bm{\theta}=\rho\bm{\mu}+\tilde{\bm{\theta}}$ where $\bm{\mu}^T\tilde{\bm{\theta}}=0$ and $\rho \in \R$.  Then the following are true
\begin{enumerate}
    \item  $f(\bm{\theta})-f(\rho\bm{\mu})\geq 0$.
    \item  $f(\bm{\theta})-f(\rho\bm{\mu})\geq 1$ provided that $\vert \rho-\rho^*\vert\leq \tfrac{1}{2}\rho^*$, $\sigma \Vert \tilde{\bm{\theta}}\Vert \geq c'$ and $\sigma \geq c\Vert \bm{\mu} \Vert$ where $c$ is defined in \eqref{eq:para_log} and \eqref{eq:para_hinge}. Here $\rho^*$ is defined in Lemma \ref{lem:minimizers} and the constant $c'$ is defined by $436$ and $8+10\rho^*\sigma^2$ for the logistic and hinge loss respectively.
\end{enumerate}
\end{lemma}

\begin{proof} We consider the logistic and hinge loss separately.
\item
\textbf{Logistic loss.} The two normal random variables, $\bm{\tilde{\theta}}^T\bm{\xi} \sim N(0,\sigma^2\Vert \bm{\tilde{\theta}} \Vert^2)$ and $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$, are independent by \eqref{eqn:fact_independence}. Since we have $\bE_{\bm{\xi}}[\log(\exp(-\bm{\tilde{\theta}}^T\bm{\xi}))]=\bE_{\bm{\xi}}[\log(\exp(\bm{\tilde{\theta}}^T\bm{\xi}))]=0$, it holds
\begin{align*}
    f(\bm{\theta})=\bE_{\bm{\xi}} \left[ \log\left (1+\exp(-\bm{\theta}^T\bm{\xi})\right)\right] &= \bE_{\bm{\xi}} \left[\log \left(1 + \exp(-\tilde{\bm{\theta}}^T\bm{\xi}) \exp(-\rho \bm{\mu}^T\bm{\xi}) \right ) \right ]\\
    &=\bE_{\bm{\xi}} \left[ \log\left (\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right]\\
    &=\bE_{\bm{\xi}} \left[ \log\left (\exp(-\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right],
\end{align*}
where the last equality is true because $\bm{\tilde{\theta}}^T\bm{\xi} \sim -\bm{\tilde{\theta}}^T\bm{\xi}$. Therefore we obtain
\begin{align*}
  \bE_{\bm{\xi}} &\left[ \log\left (1+\exp(-\bm{\theta}^T\bm{\xi})\right)\right]\\
  & \qquad \qquad= \frac{1}{2} \bE_{\bm{\xi}} \left[ \log\left (\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right] + \frac{1}{2}\bE_{\bm{\xi}} \left[ \log\left (\exp(-\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right] \\
  & \qquad \qquad= \frac{1}{2} \bE_{\bm{\xi}} \left[ \log\left ( (\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi}))(\exp(-\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})) \right)\right]\\
  &\qquad \qquad=\frac{1}{2}\bE_{\bm{\xi}} \left[\log\left(1+\exp(-\bm{\tilde{\theta}}^T\bm{\xi}-\rho \bm{\mu}^T\bm{\xi})+\exp(\bm{\tilde{\theta}}^T\bm{\xi}-\rho \bm{\mu}^T\bm{\xi})+\exp(-2\rho \bm{\mu}^T\bm{\xi})\right) \right].
  \end{align*}
By the equality $\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\bm{\tilde{\theta}}^T\bm{\xi})= 2+4\sinh^2(\tfrac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})$, we have
\begin{align*}
   \bE_{\bm{\xi}} &\left[ \log\left (1+\exp(-\bm{\theta}^T\bm{\xi})\right)\right]\\
    & \qquad \qquad \qquad =\frac{1}{2}\bE_{\bm{\xi}} \left[\log\left(1+2\exp(-\rho \bm{\mu}^T\bm{\xi})+\exp(-2\rho \bm{\mu}^T\bm{\xi})+4\sinh^2(\tfrac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\exp(-\rho \bm{\mu}^T\bm{\xi})\right) \right].
\end{align*}
Therefore, we have
\begin{equation} \begin{aligned} \label{eqn: high_noise_blah_20}
     2\bE_{\bm{\xi}}\left[\log\left( \frac{1+\exp(-\bm{\theta}^T\bm{\xi})}{1+\exp(-\rho \bm{\mu}^T\bm{\xi})}\right)\right]&=2\bE_{\bm{\xi}}\left[\log(1+\exp(-\bm{\theta}^T\bm{\xi}))\right]-\bE_{\bm{\xi}}\left[\log\left(1+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)^2\right]\\
     \qquad \qquad &= \bE_{\bm{\xi}}\left[\log \left ( 1 +  \frac{4\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\exp(-\rho \bm{\mu}^T\bm{\xi})}{(1+ \exp(-\rho \bm{\mu}^T\bm{\xi}))^2} \right ) \right] \ge 0.
     \end{aligned} \end{equation}
     Thereby, we showed that $f(\bm{\theta})-f(\rho \bm{\mu})\geq 0$. Now we establish the positive lower bound. First, we note the following $1+ \exp(-\rho \bm{\mu}^T\bm{\xi}) = 2 \exp( -\tfrac{\rho \bm{\mu}^T\bm{\xi}}{2}) \cosh(\tfrac{\rho \bm{\mu}^T\bm{\xi}}{2})$. Fix a  constant $r > 0$ and consider the set $\{\bm{\xi}: |\bm{\theta}^T\bm{\xi}| > r\}$. Applying the inequality $x^2+y^2\geq 2\vert xy \vert$ and \eqref{eqn: high_noise_blah_20}, we obtain that
     \begin{align}
   2\bE_{\bm{\xi}}\left[\log\left( \frac{1+\exp(-\bm{\theta}^T\bm{\xi})}{1+\exp(-\rho \bm{\mu}^T\bm{\xi})}\right)\right]  & = \bE_{\bm{\xi}}\left[\log\left(1 + \frac{4\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\exp(-\rho \bm{\mu}^T\bm{\xi})}{(1+ \exp(-\rho \bm{\mu}^T\bm{\xi}))^2}\right) \right] \nonumber \\
     &= \bE_{\bm{\xi}}\left[\log\left(1+\frac{\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})}{\cosh^2(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})} \right) \right] \nonumber \\
     &\geq \bE_{\bm{\xi}}\left[\log\left(1+\frac{\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})}{\cosh^2(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})} \right)\cdot1_{\{\bm{\xi}:\vert \tilde{\bm{\theta}}^T\bm{\xi}\vert \geq r\}}\right] \label{eq:high_noise_blah_21}
     \\
     &\geq \bE_{\bm{\xi}}\left[\left(\log2+\log\left( \frac{\vert\sinh(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\vert}{\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})}\right) \right)\cdot1_{\{\bm{\xi}:\vert \tilde{\bm{\theta}}^T\bm{\xi}\vert \geq r\}}\right] \nonumber.
    \end{align}
Here  \eqref{eq:high_noise_blah_21} follows from $\log\left(1+\frac{\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})}{\cosh^2(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})} \right)$ is always positive. From \eqref{eq:fact_affine}, we have $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$ and $\tilde{\bm{\theta}}^T\bm{\xi} \sim N(0,\sigma^2\Vert \tilde{\bm{\theta}} \Vert^2)$, so $\tilde{\bm{\theta}}^T\bm{\xi} = \sigma \| \tilde{\bm{\theta}} \| \psi$ where $\psi \sim N(0,1)$. Moreover, a simple computation shows that $-\log\left(\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})\right)1_{\{\vert \tilde{\bm{\theta}}^T\bm{\xi}\vert \geq r\}} \geq -\log\left(\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})\right) $ since $\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})\geq 1$ always holds. Using the inequality $\log\cosh(x)\leq \vert x\vert$ for $x$, the following bound holds
     \begin{equation} \begin{aligned} \label{eq: high_noise_22}
       &\bE_{\bm{\xi}}\left[\log\left( \frac{1+\exp(-\bm{\theta}^T\bm{\xi})}{1+\exp(-\rho \bm{\mu}^T\bm{\xi})}\right)\right]\\
       & \quad \ge \tfrac{1}{2} \log(2) \cdot \bE_{\psi}\left[1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] + \tfrac{1}{2} \bE_{\psi}\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] - \tfrac{1}{2}\bE_{\bm{\xi}} \left [ \log( \cosh(\tfrac{\rho}{2} \bm{\mu}^T\bm{\xi})) \right ]\\
       & \quad \ge \tfrac{1}{2} \log(2) \cdot \bE_{\psi}\left[1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] + \tfrac{1}{2} \bE_{\psi}\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] - \tfrac{1}{2} \bE_{\bm{\xi}} \left [ | \tfrac{\rho}{2} \bm{\mu}^T\bm{\xi} | \right ]\\
       &\quad\geq \tfrac{1}{2} \log(2) \cdot \bE_{\psi}\left[1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] + \tfrac{1}{2} \bE_{\psi}\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right]-\frac{3}{4}\left(\frac{\Vert \bm{\mu}\Vert^2}{\sigma^2}+ \sqrt{\frac{2}{\pi}}\cdot\frac{\Vert \bm{\mu} \Vert}{\sigma} \right),
    \end{aligned} \end{equation}
where the last inequality uses \eqref{fact:norm_Gaussians} and $\rho\leq \frac{3}{\sigma^2}$. Using the inequality $\vert \sinh(x) \vert \geq \exp(\frac{\vert x\vert}{2})$ for all $\vert x \vert \geq 2\log(\sqrt{2}+1)$ and letting $r=4\log(\sqrt{2}+1)$, we obtain
\begin{align}
   \tfrac{1}{2} \log(2) \cdot \bE_{\psi} \big [ 1_{\{|\psi| \ge \frac{4 \log( \sqrt{2}+ 1)\}}{\sigma \norm{\tilde{\bm{\theta}}} }} \big ] &+ \tfrac{1}{2}\bE_\psi\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{|\psi| \geq \frac{4\log(\sqrt{2}+1)}{\sigma \Vert \tilde{\bm{\theta}}\Vert}\}}\right]\nonumber\\
   &\geq \tfrac{1}{2} \log(2) \cdot \bE_{\psi} \big [ 1_{\{|\psi| \ge \frac{4 \log( \sqrt{2}+ 1)}{\sigma \norm{\tilde{\bm{\theta}}} } \}} \big ] +  \tfrac{1}{2}\bE_\psi\left[\left\vert\tfrac{\sigma \Vert \tilde{\bm{\theta}}\Vert \psi}{4} \right\vert1_{\{|\psi| \geq \frac{4\log(\sqrt{2}+1)}{\sigma \Vert \tilde{\bm{\theta}}\Vert}\}}\right]\nonumber\\
    &\geq \tfrac{1}{2} \log(2) \cdot \bE_{\psi}[1_{\{\vert\psi\vert \ge 1\}}] + \tfrac{1}{2} \bE_\psi\left[\left\vert\tfrac{\sigma \Vert \tilde{\bm{\theta}}\Vert \psi}{4} \right\vert1_{\vert\psi\vert \geq 1\}}\right] \label{eq:high_noise_blah_111} \\\nonumber
    &\geq \left ( \tfrac{1}{2} \log(2) + \frac{\sigma \Vert \tilde{\bm{\theta}}\Vert}{8} \right )\cdot\Phi^c(1).
\end{align}
Here \eqref{eq:high_noise_blah_111} follows from the assumption that $\sigma \Vert \tilde{\bm{\theta}}\Vert\geq 436
$. Combining  \eqref{eq: high_noise_22}, \eqref{eq:high_noise_blah_111} and the bounds $\sigma \geq 0.33\Vert \bm{\mu} \Vert$ and $\sigma\Vert \tilde{\bm{\theta}}\Vert \geq 436$ the result follows. \item
\textbf{Hinge loss.} We begin by denoting
$\xi_1:=\bm{\xi}^T\tilde{\bm{\theta}}$ and $\xi_2:=\bm{\xi}^T\bm{\mu}$. Notice that $\xi_1$ and $\xi_2$ are independent random variables. Recall that $\ell(t):=\ell(t,1)=\max(0,1-t)$. We have that
\begin{align*}
    f(\bm{\theta})-f(\rho \bm{\mu})&=\bE_{\bm{\xi}}\left[\ell(\bm{\xi}^T\bm{\theta})-\ell(\rho\bm{\xi}^T\bm{\mu}) \right]\\&=\bE_{\xi_1,\xi_2}\left[\ell(\xi_1+\rho\xi_2)-\ell(\rho\xi_2) \right]\\&=\bE_{\xi_1,\xi_2}\left[\ell(-\xi_1+\rho\xi_2)-\ell(\rho\xi_2) \right].
\end{align*}
The second equality follows since $\xi_1 \sim -\xi_1$. We define the function
\[
\kappa(\xi_1,\xi_2):=\ell(\xi_1+\rho\xi_2)+\ell(-\xi_1+\rho\xi_2)-2\ell(\rho\xi_2).
\]
We therefore obtain that
\[
2\left(f(\bm{\theta})-f(\rho\bm{\mu})\right)=\bE_{\xi_1,\xi_2}\left[\kappa(\xi_1,\xi_2)\right].
\]
Next we claim that
\begin{equation}\label{eq:hinge_lemma_claim1}
    \kappa(\xi_1,\xi_2)=0 \text{ whenever } \vert \xi_1\vert\leq \vert 1-\rho\xi_2\vert.
\end{equation}
To see this, suppose that $\vert \xi_1\vert\leq \vert 1-\rho\xi_2\vert$ holds. We consider two cases.  First, assume that $0\leq 1-\rho\xi_2$ which yields that $\rho\xi_2-\xi_1\leq 1$ and $\rho\xi_2+\xi_1\leq 1$. We therefore have $\kappa(\xi_1,\xi_2)=1-\xi_1-\rho\xi_2+1+\xi_1-\rho\xi_2-2(1-\rho\xi_2)=0$. Second, assume that $1-\rho\xi_2\leq 0$. It thus holds that $1\leq \rho\xi_2-\xi_1$ and $1\leq \rho\xi_2+\xi_1$. Now it immediately follows that $\kappa(\xi_1,\xi_2)=0$ and equation \eqref{eq:hinge_lemma_claim1} is established. We claim the following
\begin{equation}\label{eq:hinge_lemma_claim2}
    \kappa(\xi_1,\xi_2)=\vert \xi_1\vert-\vert 1-\rho\xi_2\vert \text{ whenever } \vert \xi_1\vert\geq \vert 1-\rho\xi_2\vert.
\end{equation}
To this end, we again consider two cases. First, assume that $\xi_1\leq -\vert 1-\rho\xi_2\vert$. This yields that $1\leq -\xi_1+\rho\xi_2$ and $\xi_1+\rho\xi_2\leq 1$, so it holds that $ \kappa(\xi_1,\xi_2)= 1-\xi_1-\rho\xi_2-2\ell(\rho\xi_2)$. The claim \eqref{eq:hinge_lemma_claim2} follows from the following simple identity
\begin{equation}\label{eq:hinge_identity}
   2\ell(t)= 1-t+\vert 1 - t \vert, \quad \forall t \in \R.
\end{equation}
Second, assume that $\xi_1\geq \vert 1-\rho\xi_2\vert$. It then holds that $\xi_1+\rho\xi_2\geq 1$ and $-\xi_1+\rho\xi_2\leq 1$ and therefore $
\kappa(\xi_1,\xi_2)=1+\xi_1-\rho\xi_2-2\ell(\rho \xi_2).
$ The claim \eqref{eq:hinge_lemma_claim2} follows from the identity \eqref{eq:hinge_identity}.  We therefore obtain
\begin{align}
    \bE_{\xi_1,\xi_2}[\kappa(\xi_1,\xi_2)]&=2\bE_{\xi_1,\xi_2}[(\ell(\xi_1+\rho\xi_2)+\ell(-\xi_1+\rho\xi_2)-2\ell(\rho\xi_2))1_{\{ \xi_1>0\}}]\label{eq:hinge_lemma_it1}\\&=2\bE_{\xi_1,\xi_2}[(\ell(\xi_1+\rho\xi_2)+\ell(-\xi_1+\rho\xi_2)-2\ell(\rho\xi_2))1_{\{ \xi_1\geq\vert 1-\rho \xi_2\vert\}}]\label{eq:hinge_lemma_it2}\\&=\bE_{\xi_1,\xi_2}[(\xi_1-\vert 1-\rho \xi_2\vert)1_{\{ \xi_1\geq \vert 1-\rho\xi_2\vert\}} ]. \label{eq:hinge_lemma_it3}
\end{align}
Here equation \eqref{eq:hinge_lemma_it1} holds because $\xi_1\sim -\xi_1$ and $\kappa(\xi_1,\xi_2)=\kappa(-\xi_1,\xi_2)$. Equation \eqref{eq:hinge_lemma_it2} is true because of claim \eqref{eq:hinge_lemma_claim1} and \eqref{eq:hinge_lemma_it3} follows from claim \eqref{eq:hinge_lemma_claim2}. From \eqref{eq:hinge_lemma_it3}, we conclude that $\bE_{\xi_1,\xi_2}[\kappa(\xi_1,\xi_2)]\geq 0$.  We then observe the bound
\begin{equation}
    \begin{aligned}
    \bE_{\xi_1,\xi_2}[(\xi_1-\vert 1-\rho \xi_2\vert)1_{\{ \xi_1\geq \vert 1-\rho\xi_2\vert\}} ] &= \tfrac{1}{2}\bE_{\xi_1,\xi_2}[\xi_1-\vert 1-\rho \xi_2\vert+\vert \xi_1-\vert 1-\rho \xi_2\vert \vert]\\&\geq -\tfrac{1}{2}\bE_{\xi_2}[\vert 1-\rho \xi_2\vert]+\tfrac{1}{2}\bE_{\xi_1,\xi_2}[\vert \xi_1\vert-\vert 1-\rho \xi_2\vert]\\&=\tfrac{1}{2}\bE_{\xi_1}[\vert \xi_1\vert]-\bE_{\xi_2}[\vert1-\rho\xi_2\vert]. \label{eq:thrid_last_equation}
\end{aligned}
\end{equation}
The second inequality follows from $\bE_{\xi_1}[\xi_1]=0$ and the triangle inequality $\vert x\vert-\vert y\vert \leq \vert \vert x\vert-y\vert$. On the other hand, it holds that
\begin{equation}\label{first_last_equation}
\bE_{\xi_1}[\vert \xi_1\vert]=\sqrt{\frac{2}{\pi}}\cdot\sigma \Vert \tilde{\bm{\theta}}\Vert,
\end{equation}
and
\begin{equation}\label{second_last_equation}
\bE_{\bm{\xi}}[\vert 1-\rho \bm{\mu}^T\bm{\xi}\vert]\leq 1+\rho \bE_{\bm{\xi}}[\vert \bm{\mu}^T\bm{\xi}\vert]\leq 1+\rho\Vert \bm{\mu} \Vert \left(\sqrt{\frac{2}{\pi}}\cdot\sigma +\Vert \bm{\mu} \Vert \right).
\end{equation}
Combing equations \eqref{eq:hinge_lemma_claim1}, \eqref{eq:hinge_lemma_claim2},  \eqref{eq:thrid_last_equation}, \eqref{first_last_equation}, and \eqref{second_last_equation}, we deduce
\begin{equation}\label{eq:last_hinge_high}
    f(\bm{\theta})-f(\rho \bm{\mu}) \geq \frac{1}{2}\left(\sqrt{\frac{1}{2\pi}}\cdot\sigma\Vert\tilde{\bm{\theta}}\Vert-1-\rho\Vert \bm{\mu} \Vert \left(\sqrt{\frac{2}{\pi}}\cdot\sigma +\Vert \bm{\mu} \Vert \right) \right).
\end{equation}
Using the bounds $\sigma\Vert \tilde{\bm{\theta}}\Vert \geq 8+10\rho^*\sigma^2$, $\sigma \geq 0.62\Vert \bm{\mu} \Vert$ and $\rho\leq \tfrac{3}{2}\rho^*$, the result follows from \eqref{eq:last_hinge_high}.
\end{proof}
We next derive a lower bound \eqref{eq:quantity_split}, Part (b). But, first we need a basic lemma from convex analysis.
\begin{lemma}\label{lem:convex_analysis_lemma}
Suppose that $g:\R_{\geq 0}\to \R$ is a convex function with a minimizer at $\rho^*>0$. Assume that $g$ is twice differentiable on the interval $[\tfrac{3}{4}\rho^*,\tfrac{5}{4}\rho^*]$ and there exists a constant $B>0$ such that $g''(\rho)\geq B$ for all $\rho \in [\tfrac{3}{4}\rho^*,\tfrac{5}{4}\rho^*]$. Then it holds that
\begin{equation}\label{eq0:cvx_lemma}
    g(\rho)-g(\rho^*)\geq \frac{\rho^*B}{8}\vert \rho-\rho^*\vert \quad \text{for all $\rho \not\in [\tfrac{1}{2}\rho^*,\tfrac{3}{2}\rho^*]$}.
\end{equation}
\end{lemma}
\begin{proof}
 The proof follows by considering the second order Taylor series expansion of the function $g$.
\end{proof}

\begin{lemma}(Lower bound for (b) in \eqref{eq:quantity_split})\label{lem:lower_bound_2} Fix $\bm{\theta}\in \R^d$ and orthogonally decompose $\bm{\theta}=\rho\bm{\mu}+\tilde{\bm{\theta}}$. Suppose that $\vert \rho-\rho^*\vert \geq \tfrac{1}{2}\rho^*$. Then provided that
$\sigma \geq c\Vert \bm{\mu} \Vert$ where the constant $c$ is defined in \eqref{eq:para_log} and \eqref{eq:para_hinge}, there exists a positive constant $A$ such that the following is true
\begin{equation}\label{eq:lemma9}
    f(\rho \bm{\mu})-f(\bm{\theta^*}) \geq A\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2}.
\end{equation}
\begin{proof} We consider the logistic and hinge loss separately.
\item \textbf{Logistic loss.} Define the function
\[
    g(\rho):=\bE_{\bm{\xi}}\left[\log\left(1+\exp(-\rho \bm{\mu}^T\bm{\xi})\right) \right], \quad \bm{\xi} \sim N(\bm{\mu},\sigma^2I_d).
\]
By Lemma~\ref{lem:minimizers}, we know that $g$ is a convex function with a unique minimizer at $\rho^*:=\frac{2}{\sigma^2}$. Observe that $f(\rho \bm{\mu})-f(\bm{\theta}^*) = g(\rho)-g(\rho^*)$; hence in order to prove \eqref{eq:lemma9}, we instead aim to bound this difference in the function $g$. From \eqref{eq:fact_affine}, we have $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$. It thus holds
\[
 4g''(\rho)=\bE\left(\frac{(\bm{\mu}^T\bm{\xi})^2}{\cosh(\frac{\rho}{2} \bm{\mu}^T\bm{\xi})^2} \right)=\frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi }}\int_{-\infty}^{\infty}\frac{z^2}{\cosh^2(\frac{\rho z}{2})}\exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2}\right)dz.
\]
Upper bounding $\cosh^2(\frac{\rho z}{2})$ by $\exp(\vert \rho z\vert)$, we next obtain
\begin{align*}
    4g''(\rho)&\geq \frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi}} \int_{-\infty}^{\infty} z^2\exp\left(-\vert \rho z \vert \right)\exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu}\Vert^2}\right)dz & \\ &=\frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi }} \int_{-\infty}^{\infty} z^2\exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2+2\sigma^2\Vert \bm{\mu}\Vert^2\vert\rho z\vert}{2\sigma^2\Vert \bm{\mu}\Vert^2}\right)dz  ,&\\
    &=\frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi }}\cdot\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right) \int_{-\infty}^{\infty} z^2\exp\left(-\frac{z^2-2\Vert \bm{\mu} \Vert^2 z+2\sigma^2\Vert \bm{\mu}\Vert^2\vert\rho z\vert}{2\sigma^2\Vert \bm{\mu}\Vert^2}\right)dz
    \\&= \frac{\sigma^2\Vert \bm{\mu}\Vert^2}{\sqrt{2\pi }}\cdot\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right)\int_{-\infty}^{+\infty} z^2\exp\left(-\frac{z^2-2\frac{\Vert \bm{\mu} \Vert}{\sigma}z+2\vert \rho z \vert \sigma\Vert\bm{\mu}\Vert }{2} \right) dz\\&\geq \frac{\sigma^2\Vert \bm{\mu}\Vert^2}{\sqrt{2\pi }}\cdot\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right)\int_{0}^{+\infty} z^2\exp\left(-\frac{z^2}{2}\right)\exp\left(z\left(\frac{\Vert\bm{\mu}\Vert}{\sigma}-\rho \sigma \Vert \bm{\mu} \Vert\right) \right)dz\\&\geq \frac{\sigma^2\Vert \bm{\mu}\Vert^2}{\sqrt{2\pi }}\cdot\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}-\frac{1}{2}-\left \vert \frac{\Vert \bm{\mu} \Vert}{\sigma}-\rho \sigma \Vert \bm{\mu} \Vert \right\vert  \right)\int_{0}^1 z^2 dz.
    \end{align*}
Here the second to last inequality follows from the change of variables $z\rightarrow z\sigma\Vert \bm{\mu}\Vert$. The last inequality follows from restricting the integral's domain to $[0,1]$ and also lower bounding $-\frac{z^2}{2}$ and $z\left(\frac{\Vert\bm{\mu}\Vert}{\sigma}-\rho \sigma \Vert \bm{\mu} \Vert\right)$ by $\frac{-1}{2}$ and $-\left\vert\frac{\Vert\bm{\mu}\Vert}{\sigma}-\rho \sigma \Vert \bm{\mu} \Vert\right\vert$ respectively. We see that, for $\rho \in [\tfrac{3}{4}\rho^*,\tfrac{5}{4}\rho^*]$,  the term $\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}-\frac{1}{2}-\left \vert \frac{\Vert \bm{\mu} \Vert}{\sigma}-\rho \sigma \Vert \bm{\mu} \Vert \right\vert  \right)$ is lower bounded by $\exp\left(-\frac{1}{2c^2}-\frac{1}{4c}-\frac{1}{2} \right)$. By Lemma \ref{lem:convex_analysis_lemma}, the result follows with the constant $A$ computed as follows
\begin{equation*}\label{eq:constant_A_log}
    A=\frac{1}{12\sqrt{2\pi}}\cdot\exp\left(-\frac{1}{2c^2}-\frac{1}{4c}-\frac{1}{2} \right).
\end{equation*}
\item \textbf{Hinge loss.} We begin by defining the function $h(\rho)=f(\rho\bm{\mu})$. Therefore
\[
f(\rho \bm{\mu})=\bE_{\bm{\xi}}[\ell(\rho\bm{\xi}^T\bm{\mu})]=\bE_{\bm{\xi}}[(1-\rho \bm{\xi}^T\bm{\mu})1_{\{\rho\bm{\xi}^T\bm{\mu}\leq 1 \}}].
\]
Hence, it holds that
\[
h'(\rho)=\bm{\mu}^T\nabla f(\rho \bm{\mu})=-\bE_{\bm{\xi}}[\bm{\xi}^T\bmu1_{\{ \rho\bm{\xi}^T\bm{\mu}\leq 1\}}].
\]
From \eqref{eq:fact_affine}, we obtain that $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$.  For $\rho>0$, therefore, it holds that
\begin{equation}\label{der_phi_pos}
    h'(\rho)=\frac{-1}{\sigma\Vert \bm{\mu} \Vert\sqrt{2\pi}}\int_{-\infty}^{\frac{1}{\rho}}z\exp\left(-\frac{1}{2}\cdot\left(\frac{z}{\sigma\Vert \bm{\mu} \Vert}-\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)^2\right)dz.
\end{equation}
Applying chain rule thus yields
\[
h''(\rho)=\frac{1}{\rho^3\sigma\Vert \bm{\mu} \Vert\sqrt{2\pi}}\exp\left(-\frac{1}{2}\cdot \left(\frac{1}{\rho\sigma\Vert \bm{\mu} \Vert}-\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)^2 \right)\quad \text{for all $\rho>0$}.
\]
Hence, for all $\rho \in [\tfrac{3}{4}\rho^*,\tfrac{5}{4}\rho^*]$ it holds that
\[
h''(\rho)\geq \frac{64}{125{\rho^*}^3\sigma\Vert \bm{\mu} \Vert\sqrt{2\pi}}\exp\left(-\frac{1}{2}\cdot \Gamma^2 \right),
\]
where $\Gamma:=\max\left\{\left|\frac{4}{3\rho^*\sigma\Vert \bm{\mu} \Vert}-\frac{\Vert \bm{\mu} \Vert}{\sigma} \right|,\left|\frac{4}{5\rho^*\sigma\Vert \bm{\mu} \Vert}-\frac{\Vert \bm{\mu} \Vert}{\sigma} \right| \right\}$. Therefore, by Lemma \ref{lem:convex_analysis_lemma} and $\vert \rho-\rho^*\vert \geq \tfrac{1}{2}\rho^*$,  it holds that
\begin{equation}\label{eq:constant_r_hinge_loss}
\begin{aligned}
    f(\rho\bm{\mu})-f(\bm{\theta}^*)&\geq \frac{4}{125\sqrt{2\pi}}\cdot\frac{\sigma}{r\Vert \bm{\mu} \Vert}\cdot\exp\left(-\frac{1}{2}\cdot \Gamma^2 \right).
\end{aligned}
\end{equation}
Here $r=\rho^*\sigma^2$. Note that $r>0$ by Lemma \ref{lem:minimizers}. We aim to lower bound the right-hand side of \eqref{eq:constant_r_hinge_loss}. We denote by $w=\frac{\sigma}{r\Vert \bm{\mu} \Vert}-\frac{\Vert \bm{\mu} \Vert}{\sigma}$ the quantity defined in Lemma \ref{lem:minimizers}. In particular, by Lemma \ref{lem:minimizers}, the following holds
\begin{equation}\label{eq:w_hinge_later}
\frac{1}{\sqrt{2\pi}}\cdot\frac{\sigma}{\Vert \bm{\mu} \Vert}=\Phi(w)\cdot \exp(\tfrac{1}{2}w^2).
\end{equation}

We consider two cases.  First suppose that $w\geq \frac{1}{(3\sqrt{2}-4)c}$. Along with the assumption $\frac{\sigma}{\Vert \bm{\mu} \Vert}\geq c$ this implies that $w\geq \frac{1}{3\sqrt{2}-4}\cdot \frac{\Vert \bm{\mu} \Vert}{\sigma}$. A simple computation shows that $w^2\geq \frac{1}{2}\cdot \Gamma^2$ for all $w\geq \frac{1}{3\sqrt{2}-4}\cdot \frac{\Vert \bm{\mu} \Vert}{\sigma}$. On the other hand, by \eqref{eq:w_hinge_later} for $w\geq 0$, we obtain that $\frac{2}{\pi}\cdot \frac{\sigma^2}{\Vert \bm{\mu} \Vert^2}\geq \exp(w^2)$. Plugging in the bounds  $w^2\geq \frac{1}{2}\cdot \Gamma^2 $,  $\exp(-w^2)\geq \frac{\pi}{2}\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2}$, and $\frac{\sigma}{r\Vert \bm{\mu} \Vert}\geq w\geq \frac{1}{(3\sqrt{2}-4)c}$ into the right-hand-side of \eqref{eq:constant_r_hinge_loss}, we obtain that
\begin{equation*}
    f(\rho\bm{\mu})-f(\bm{\theta}^*)\geq \frac{\sqrt{2\pi}}{125(3\sqrt{2}-4)c}\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2}.
\end{equation*}
Next, suppose that $w<\frac{1}{(3\sqrt{2}-4)c}$. In this case, the two factors $\frac{\sigma}{r\Vert \bm{\mu} \Vert}$ and $\exp\left(-\frac{1}{2}\cdot\Gamma^2 \right)$ in \eqref{eq:constant_r_hinge_loss} are lower bounded separately. Note that it always holds that $w\geq -\frac{\Vert \bm{\mu} \Vert}{\sigma}$ as $r>0$. Therefore, it is easy to see that the latter factor is lower bounded by $\exp\left(-\frac{1}{2} \left(\frac{4}{3(3\sqrt{2}-4)c}+\frac{1}{3c} \right)^2\right)$. Hence, it remains to bound the factor $\frac{\sigma}{r\Vert \bm{\mu} \Vert}$ in \eqref{eq:constant_r_hinge_loss}. To this end, we show that $w\geq -\frac{\Vert \bm{\mu} \Vert}{2\sigma}$ for all $\frac{\sigma}{\Vert \bm{\mu} \Vert}\geq c$.  Note that a chain of change of variables gives
  \begin{equation*}\label{eq:myeq_blah}
  \begin{aligned}
      \Phi(w)\cdot\exp\left(\frac{w^2}{2}\right)&=\frac{1}{\sqrt{2\pi}}\cdot\int_{0}^{+\infty}\exp(-\frac{1}{2}t^2)\cdot\exp(wt) \, dt.
 \end{aligned}\end{equation*}
The right-hand side of \eqref{eq:w_hinge_later} is an increasing function with respect to $w$. Therefore it suffices to show that the following holds
\begin{equation}\label{eq:sigma/bmu>c}
    \frac{1}{\sqrt{2\pi}}\cdot \frac{\sigma}{\Vert \bm{\mu} \Vert} \geq \Phi\left(-\frac{\Vert \bm{\mu} \Vert}{2\sigma} \right)\cdot \exp\left(\frac{\Vert \bm{\mu} \Vert^2}{8\sigma^2} \right) \quad \text{whenever} \quad \frac{\sigma}{\Vert \bm{\mu} \Vert}\geq c.
\end{equation}
However, it can be verified by a plot that $\tfrac{1}{\sqrt{2\pi}}\geq t\cdot \Phi\left(-\tfrac{t}{2} \right)\cdot \exp\left(\frac{t^2}{8} \right)$ holds for all $t\in (0,\frac{1}{c})$. Therefore, we have shown that $w\geq -\frac{\Vert \bm{\mu} \Vert}{2\sigma}$ which implies that $\frac{\sigma}{r\Vert \bm{\mu} \Vert}\geq \frac{\Vert \bm{\mu} \Vert}{2\sigma}$. Finally we lower bound the quantity $\frac{\sigma}{r\Vert\bm{\mu}\Vert}$ by $c\cdot\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}$. We have concluded \eqref{eq:lemma9} in case of hinge loss function where the constant $A$ can be computed as follows
\begin{equation*}\label{eq:constant_A_hinge}
    A=\min\left\{\frac{c}{2}\cdot\exp\left(-\frac{1}{2} \left(\frac{4}{3(3\sqrt{2}-4)c}+\frac{1}{3c} \right)^2\right), \frac{\sqrt{2\pi}}{125(3\sqrt{2}-4)c} \right\}.
\end{equation*}
\end{proof}
\end{lemma}
We now have the ingredients to prove Theorem \ref{thm:high}.
\begin{proof}[Proof of Theorem \ref{thm:high}]
Consider the set $C$ and function $V$ defined in \eqref{eq:CV_high_recall}:
\begin{equation}\label{eq:CV_high_proofthm3}
    C:=\left\{\bm{\theta}: \vert \rho -\rho^*\vert<\tfrac{1}{2}\rho^* \text{ and } \sigma \Vert \tilde{\bm{\theta}}\Vert \leq c'\right\} \quad \text{ and }\quad  V(\bm{\theta})=\frac{1}{2\alpha}\Vert \bm{\theta}-\bm{\theta}^*\Vert^2.
\end{equation}
We let $c'$ to be defined as in Lemma \ref{lem:lower_bound_1}. This means that $c'$ equals to $436$ and $8+10\rho^*\sigma^2$ in case of logistic and hinge loss respectively. We next show that there exists a positive constant $\delta$ such that the following is true
\begin{equation}
    \bP_{\bm{\xi}}\left(\bm{\xi}^T\bm{\theta}\geq 1\right)\geq \delta \quad \text{for all}\quad \bm{\theta} \in C.
    \end{equation}
Let $\bm{\theta}\in C$ and orthogonally decompose it into $\bm{\theta}=\rho\bm{\mu}+\tilde{\bm{\theta}}$. We have that $\bm{\xi}^T\bm{\theta}=\rho\bm{\xi}^T\bm{\mu}+\bm{\xi}^T\tilde{\bm{\theta}}$. Note that $\rho>0$ as $\bm{\theta}\in C$. By \eqref{eqn:fact_independence}, we see that $\bm{\xi}^T\bm{\theta}$ and $\bm{\xi}^T\tilde{\bm{\theta}}$ are independent normal random variables. It thus holds that
\begin{equation}
    \bP_{\bm{\xi}}\left(\bm{\xi}^T\bm{\theta}\geq 1\right)\geq \bP_{\bm{\xi}}\left(\rho\bm{\xi}^T\bm{\mu}\geq 1\right)\cdot \bP_{\bm{\xi}}\left(\bm{\xi}^T\tilde{\bm{\theta}}\geq 0\right) =\frac{1}{2}\cdot\bP_{\bm{\xi}}\left(\bm{\xi}^T\bm{\mu}\geq \frac{1}{\rho}\right).
\end{equation}
Rewrite the inequality $\bm{\xi}^T\bm{\mu}\geq \frac{1}{\rho}$ by $z:=\frac{\bm{\xi}^T\bm{\mu}-\Vert \bm{\mu} \Vert^2}{\sigma\Vert \bm{\mu} \Vert}\geq \frac{\frac{1}{\rho}-\Vert \bm{\mu} \Vert^2}{\sigma\Vert
\bm{\mu} \Vert}$. Noting that $z\sim N(0,1)$ and using the inequality $\frac{2}{\rho^*}\geq\frac{1}{\rho}$, we obtain that
\begin{equation}\label{eq:delta_proof}
    \bP_{\bm{\xi}}\left(\bm{\xi}^T\bm{\theta}\geq 1\right)\geq \delta:= \frac{1}{2}\cdot\Phi^c\left(\frac{\frac{2}{\rho^*}-\Vert\bm{\mu} \Vert^2}{\sigma\Vert \bm{\mu} \Vert}\right).
\end{equation}
We next show that the pair $(C,V)$ satisfies the drift equation \eqref{eq:driftequation}.  Let us rewrite \eqref{eq:quantity_split}:
\begin{equation}\label{eq:quantity_split2}
\begin{aligned}
f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)= & \underbrace{f(\bm{\theta}_{k-1})-f(\rho_{k-1}\bm{\mu})}_{(a)}+\underbrace{f(\rho_{k-1}\bm{\mu})-f(\bm{\theta}^*)}_{(b)}.
\end{aligned}
\end{equation}
By Lemmas \ref{lem:lower_bound_1} and \ref{lem:lower_bound_2}, both terms in $(a)$ and $(b)$ in \eqref{eq:quantity_split2} are non-negative . Assume that $\bm{\theta}_{k-1}\not\in C$. Therefore, either $\sigma \Vert \tilde{\bm{\theta}}_{k-1}\Vert\geq c'$ or $\vert\rho_{k-1}-\rho^*\vert\geq \tfrac{1}{2}\rho^*$; this implies that the quantity $(a)$ is at least 1 or the quantity $(b)$ is at least $A\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2}$ respectively. The constant $A$ in Lemma \ref{lem:lower_bound_2} satisfies $1\geq A\cdot \frac{\Vert \bm{\mu} \Vert^2}{\sigma^2}$ for all $\frac{\sigma}{\Vert \bm{\mu} \Vert}\geq c$. Hence it holds that
\begin{equation}
    A\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2}\leq f(\bm{\theta}_{k-1})-f(\bm{\theta}^*) \quad \text{for all } \bm{\theta}_{k-1}\not\in C.
\end{equation}
We use \eqref{eq:high_convex_tech_lemma} next to establish the drift equation \eqref{eq:driftequation}. Recall that the following holds
    \begin{equation}
       f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)\leq \frac{1}{2\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta} ^*\Vert^2-\bE\left[\Vert \bm{\theta}_{k}-\bm{\theta}^* \Vert^2\, |\mathcal{F}_{k-1}\right] \right)+\frac{\alpha}{2}\left(\Vert \bm{\mu} \Vert^2+d\sigma^2\right).
    \end{equation}
Combining the last two displayed inequalities and using the definition of function $V$, we obtain that
\begin{equation}
     \left(\bE\left[V(\bm{\theta}_k)|\mathcal{F}_{k-1}\right]-V(\bm{\theta}_{k-1})\right)\cdot 1_{\{\bm{\theta}_{k-1}\not\in C\}} \leq \left(\frac{\alpha}{2}(\Vert \bm{\mu} \Vert^2+d\sigma^2)-A\cdot\frac{\Vert \bm{\mu}\Vert^2}{\sigma^2}\right)\cdot 1_{\{\bm{\theta}_{k-1}\not\in C\}}.
\end{equation}
Therefore, by choosing $\alpha<A\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2(\Vert \bm{\mu} \Vert^2+d\sigma^2)}$, we obtain the drift equation \eqref{eq:driftequation} holds with $b:=\frac{A}{2}\cdot\frac{\Vert \bm{\mu} \Vert^2}{\sigma^2}$. Next, we obtain bounds on $\bE[\tau_m]$ for $m\geq 1$. By Lemma \ref{lem:drift_from_meyn} and a simple induction, we obtain that
 \begin{equation}
      \bE[\tau_m]\leq \tfrac{1}{b}V(0)+\tfrac{1}{b}(m-1)\sup_{\bm{\theta}\in C} V(\bm{\theta}).
 \end{equation}
 Compactness of set $C$ yields that, $\sup_{\bm{\theta}\in C} V(\bm{\theta})<+\infty$. Therefore, for some constant $\gamma$, the following is true
 \begin{equation}\label{eq:bound_on_tau_m_high}
      \bE[\tau_m] \leq \gamma\cdot m.
 \end{equation}
Combining \eqref{eq:bound_on_tau_m_high}, \eqref{eq:delta_proof} and Lemma \ref{lem:ETleqET_C}, the proof immediately follows.
\end{proof}
\subsection{Hinge Regression}
In this section, we study the following optimization problem
\begin{align*}
   \min f(\bm{\theta}):=\bE_{\bm{\xi} \sim \mathcal{P}_*} [\ell(\bm{\xi}^T\bm{\theta})],
\end{align*}
where $\ell(x):=\ell(x,1)$ with the hinge loss function defined in \eqref{eq:hinge_loss_definition}.  We state the main theorem of this section below. We defer the proof to the end of this subsection.

\begin{theorem}\label{theorem_hinge}
Let $\bm{\theta}_0=\bm{0}$. The following are true.
\begin{enumerate}
    \item  Fix $\epsilon\in (0,\tfrac{1}{2})$ and let $M=1+\frac{\alpha(\Vert \bm{\mu} \Vert^2+\sigma^2)}{2\epsilon}$. Suppose that $\sigma \leq \left(\tfrac{1}{2}-\epsilon\right)\sqrt{\tfrac{\pi}{2}}\Vert \bm{\mu} \Vert$ (low variance regime). The following then holds.
\begin{equation}\begin{aligned} \label{eq:low_Et_proof}
\bE[T]&\leq \left(2+\frac{\left(M-1\right)^2}{\alpha\Vert \bm{\mu} \Vert^2}+\frac{\sigma}{\Vert \bm{\mu} \Vert}\left(\frac{1.6M^2}{\alpha \Vert \bm{\mu} \Vert^2}+7,612\sigma^2\alpha^4\Vert\bm{\mu} \Vert^4\right) \right)+\frac{2M^2}{\alpha \Vert\bm{\mu}\Vert^2}.
\end{aligned} \end{equation}
\item Suppose that $\sqrt{\frac{\pi}{8}}\Vert \bm{\mu} \Vert \leq \sigma$ (high variance regime).  It holds that $\bE[T]<+\infty$ provided the step-size $\alpha$ satisfies
\begin{equation}
    \alpha< \frac{\min\{1,\tfrac{1}{2}r\rho^*\}}{\Vert \bm{\mu} \Vert^2+d\sigma^2}
\end{equation}
where the constant $r$ is defined as
\begin{equation*}
    r:= \frac{64}{125 \rho^*\sigma \Vert \bm{\mu} \Vert \sqrt{2\pi}}\exp\left(-\frac{(\tfrac{4}{3}\frac{1}{\rho^*}-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2} \right).
\end{equation*}
\end{enumerate}
  \end{theorem}

\subsubsection{Low Variance Regime}\label{sec:low_hinge}
Throughout this section, we fix a positive constant $0<\epsilon<\tfrac{1}{2}$ and consider the following drift function.
\[
V(\bm{\theta}):=\frac{1}{\alpha \Vert \bm{\mu}\Vert^2}(M-\bm{\mu}^T\bm{\theta})^2,
\]
where $M=1+\frac{\alpha(\Vert \bm{\mu} \Vert^2+\sigma^2)}{2\epsilon}$. In addition, we consider the following target set.
\begin{equation}\label{eq:hinge_low_C}
    C:=\{\bm{\theta}:\bm{\mu}^T\bm{\theta} \geq 1\}.
\end{equation}
\begin{proposition}\label{prop:low_noise_hinge_Etau1}
Fix $\epsilon\in(0,\tfrac{1}{2})$ and suppose that $\sigma \leq \left(\tfrac{1}{2}-\epsilon\right)\sqrt{\tfrac{\pi}{2}}\Vert \bm{\mu} \Vert$. Let $\bm{\theta}\in \R^d$ such that $\bm{\mu}^T\bm{\theta}\leq 1$. The following is then true
\begin{equation}
    \bE[\tau_1^C|\bm{\theta}_0=\bm{\theta}]\leq V(\bm{\theta}).
\end{equation}
\end{proposition}
\begin{proof}

Suppose that $\bm{\mu}^T\bm{\theta}_{k-1}\leq 1$. It, then, holds that
\begin{align}
    \bE[(M-\bm{\mu}^T\bm{\theta}_k)^2|\mathcal{F}_{k-1}]1_{\{\tau_1^C\wedge n\geq k \}}&= \bE_{\bm{\xi}_k}[(M-\bm{\mu}^T\bm{\theta}_{k-1}-\alpha 1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\bm{\mu}^T\bm{\xi}_k)^2|\mathcal{F}_{k-1}]1_{\{\tau_1^C\wedge n\geq k \}}\\&=(M-\bm{\mu}^T\bm{\theta}_{k-1})^2-2\alpha(M-\bm{\mu}^T\bm{\theta}_{k-1})\bE_{\bm{\xi}_k}[1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\bm{\mu}^T\bm{\xi}_k|\mathcal{F}_{k-1}]\\&+\alpha^2\bE_{\bm{\xi}_k}[1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}(\bm{\mu}^T\bm{\xi}_k)^2|\mathcal{F}_{k-1}].
\end{align}
We need to lower bound the quantity $\bE_{\bm{\xi}_k}[1_{\{ \bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\bm{\mu}^T\bm{\xi}_k|\mathcal{F}_{k-1}]$. We have
\begin{align}
    \bE_{\bm{\xi}_k}[1_{\{ \bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\bm{\mu}^T\bm{\xi}_k|\mathcal{F}_{k-1}]&=\Vert \bm{\mu}\Vert^2\bE_{\bm{\xi}_k}[1_{\{ \bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}|\mathcal{F}_{k-1}]+\bE_{\bm{\xi}_k}[1_{\{ \bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\bm{\mu}^T(\bm{\xi}_k-\bm{\mu})|\mathcal{F}_{k-1}]\\&\geq \tfrac{1}{2}\Vert \bm{\mu} \Vert^2-\bE_{\bm{\xi}_k}[\vert \bm{\mu}^T(\bm{\xi}_k-\bm{\mu})]\vert]\\&\geq \tfrac{1}{2}\Vert \bm{\mu} \Vert^2-\sqrt{\tfrac{2}{\pi}}\sigma\Vert \bm{\mu} \Vert.
\end{align}
Here we used that $1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq \bm{\mu}^T\bm{\theta}_{k-1}\}}\leq 1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}$ and  $\bE_{\bm{\xi}_k}[1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq \bm{\mu}^T\bm{\theta}_{k-1}\}}]=\tfrac{1}{2}$. Continuing, we obtain that
\begin{align*}
\bE[(M-\bm{\mu}^T\bm{\theta}_k)^2-(M-\bm{\mu}^T\bm{\theta}_{k-1})^2&|\mathcal{F}_{k-1}]1_{\{\tau_1^C\wedge n\geq k \}}\\
&\leq \alpha^2 \bE_{\bm{\xi}_k}[(\bm{\mu}^T\bm{\xi}_k)^2]-2\alpha(M-\bm{\mu}^T\bm{\theta}_{k-1})(\tfrac{1}{2}\Vert \bm{\mu} \Vert^2-\sqrt{\tfrac{2}{\pi}}\sigma\Vert \bm{\mu} \Vert)\\&\leq \alpha^2\Vert \bm{\mu}\Vert^2(\Vert \bm{\mu} \Vert^2+\sigma^2)-2\alpha(M-\bm{\mu}^T\bm{\theta}_{k-1})\left(\tfrac{1}{2}\Vert \bm{\mu} \Vert^2-\sqrt{\tfrac{2}{\pi}}\sigma\Vert \bm{\mu} \Vert\right)
\end{align*}
Hence, using the bounds $
\sigma\leq \left(\tfrac{1}{2}-\epsilon\right)\sqrt{\tfrac{\pi}{2}}\Vert \bm{\mu} \Vert$ and $M\geq 1+\frac{\alpha(\Vert \bm{\mu} \Vert^2+\sigma^2)}{2\epsilon}$, we obtain that
\begin{align*}
    \bE[V(\bm{\theta}_k)-V(\bm{\theta}_{k-1})|\mathcal{F}_{k-1}]1_{\{\tau_1^C\wedge n\geq k \}}\leq \alpha (\Vert \bm{\mu} \Vert^2+\sigma^2)-2(M-\bm{\mu}^T\bm{\theta}_{k-1})\left(\tfrac{1}{2}-\sqrt{\tfrac{2}{\pi}}\frac{\sigma}{\Vert \bm{\mu} \Vert}\right)\leq -1_{\{\tau_1^C\wedge n\geq k \}}.
\end{align*}

Using Appendix Lemma \ref{lem:drift_lemma_appendix}, we obtain that $\bE[\tau_1^C\wedge n|\mathcal{F}_{-1}] \leq V(\bm{\theta})$. Monotone convergence theorem completes the proof.
\end{proof}
We next upper bound the expected value of $\tau_m^C$ when we initialize with $\bm{\theta}_0=\bm{0}$.
\textcolor{red}{Is the proof of this proposition the same as Proposition 2 in logistic}
\textcolor{blue}{Sina:They have a lot in common! but not entirely the same. It would be best if we make this one shorter.}
\begin{proposition}\label{prp:hinge_low_etm}
Let $\bm{\theta}_0=\bm{0}$ and suppose that $\sigma\leq \left(\tfrac{1}{2}-\epsilon\right)\sqrt{\tfrac{\pi}{2}}\Vert \bm{\mu} \Vert$. Consider the set $C$ defined in \eqref{eq:hinge_low_C}. Then the following bound holds for all $m \ge 1$
\begin{equation}
   \bE[\tau_m^C]\leq (m-1)\left(2+\frac{\left(M-1\right)^2}{\alpha\Vert \bm{\mu} \Vert^2}+\frac{\sigma}{\Vert \bm{\mu} \Vert}\left(\frac{1.6M^2}{\alpha \Vert \bm{\mu} \Vert^2}+7,612\sigma^2\alpha^4\Vert\bm{\mu} \Vert^4\right) \right)+\frac{M^2}{\alpha \Vert\bm{\mu}\Vert^2},
\end{equation}
where $M=1+\frac{\alpha(\Vert \bm{\mu} \Vert^2+\sigma^2)}{2\epsilon}$.
\end{proposition}

\begin{proof}
Similar to \eqref{eq: low_noise_bound}, we obtain that
\begin{equation}\label{eq:low_noise_hinge1}
    \bE\left[(\tau_m^{C}-\tau_{m-1}^{C})\wedge n|\mathcal{F}_{\tau_{m-1}^{C}+1}\right]\leq 2+\sum_{i=1}^{\infty}\bE\left[\tau_1^{C}\wedge n|\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}^{C}+1}\right]1_{\{i-1\leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<i\}}.
\end{equation}
For each $i \ge 2$, we observe the bound
\begin{align*}
    i-1 \leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1} &=1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}}-\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^C+1}1_{\{\bm{\xi}_{\tau_{m-1}^C+1}^T\bm{\theta}_{\tau_{m-1}^C}\leq 1\}}\\&\leq -\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^C+1}1_{\{\bm{\xi}_{\tau_{m-1}^C+1}^T\bm{\theta}_{\tau_{m-1}^C}\leq 1\}}\\&\leq -\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^C+1}.
    \end{align*}
    Here the first inequality follows since $1\leq \bm{\mu}^T\bm{\theta}_{\tau_{m-1}^C}$ and the second inequality from the fact that $0\leq -\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^C+1}$ as $i-1> 0$. We next, similar to our computation in \eqref{eq: low_noise_blah_3} and \eqref{eq: low_noise_blah_4}, have that
    \begin{equation} \begin{aligned}  \label{eq: low_noise_hinge2}
\bE\left[\tau_1^{C}\wedge n|\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}^{C}+1}\right]1_{\{i-1\leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<i\}}&\leq  \frac{(M+i-1)^2}{\alpha \Vert \bm{\mu}\Vert^2}1_{\{ \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^{C}+1}\leq \frac{1-i}{\alpha}\}} \quad \forall i\geq 2.
\end{aligned} \end{equation}
Moreover,
\begin{equation} \begin{aligned} \label{eq: low_noise_hinge_3}
    \bE\left[1_{\{ \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^{C}+1}\leq \frac{1-i}{\alpha}\}} \right]&=\Phi\left(\frac{\frac{1-i}{\alpha}-\Vert \bm{\mu}\Vert^2}{\sigma\Vert \bm{\mu}\Vert } \right).
\end{aligned} \end{equation}
Combining \eqref{eq:low_noise_hinge1}, \eqref{eq: low_noise_hinge2} and \eqref{eq: low_noise_hinge_3} we obtain that
\begin{equation}\label{eq:low_noise_hinge_blah12}
    \bE\left[(\tau_m^{C}-\tau_{m-1}^{C})\wedge n|\mathcal{F}_{\tau_{m-1}^{C}+1}\right]\leq 2+\frac{\left(M-1\right)^2}{\alpha\Vert \bm{\mu} \Vert^2}+\frac{\sigma}{\Vert \bm{\mu} \Vert\sqrt{2\pi}}\sum_{i=2}^{+\infty}\frac{\left(M+i-1\right)^2}{\alpha \Vert \bm{\mu} \Vert^2+i-1}\exp\left(-\tfrac{1}{2}\left(\frac{\Vert \bm{\mu} \Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu} \Vert^2}\right)^2 \right).
\end{equation}
Note that we used the bound $\bE\left[\tau_1^{C}\wedge n|\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}^{C}+1}\right]1_{\{0\leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<1\}}\leq \frac{(M-1)^2}{\alpha \Vert\bm{\mu}\Vert^2}$. Continuing, similar to \eqref{eq: low_noise_blah_6} and \eqref{eq: low_noise_blah_7}, we have
\begin{equation}
    \sum_{i=3}^{+\infty}\frac{\left(M+i-1\right)^2}{\alpha \Vert \bm{\mu} \Vert^2+i-1}\exp\left(-\tfrac{1}{2}\left(\frac{\Vert \bm{\mu} \Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu} \Vert^2}\right)^2 \right) \leq 19,080 \alpha^4 \sigma^4 \Vert \bm{\mu} \Vert^4.
\end{equation}
We upper bound the term in \eqref{eq:low_noise_hinge_blah12} for $i=2$ by $\frac{4M^2}{\alpha \Vert \bm{\mu} \Vert^2}$. Finally we conclude that
\begin{equation}
    \bE\left[\tau_m^C-\tau_{m-1}^C\wedge n \right] \leq 2+\frac{\left(M-1\right)^2}{\alpha\Vert \bm{\mu} \Vert^2}+\frac{\sigma}{\Vert \bm{\mu} \Vert}\left(\frac{1.6M^2}{\alpha \Vert \bm{\mu} \Vert^2}+7,612\sigma^2\alpha^4\Vert\bm{\mu} \Vert^4\right).
\end{equation}
We, therefore, obtain that
\begin{equation}
    \bE\left[\tau_m^C\right] \leq (m-1)\left(2+\frac{\left(M-1\right)^2}{\alpha\Vert \bm{\mu} \Vert^2}+\frac{\sigma}{\Vert \bm{\mu} \Vert}\left(\frac{1.6M^2}{\alpha \Vert \bm{\mu} \Vert^2}+7,612\sigma^2\alpha^4\Vert\bm{\mu} \Vert^4\right) \right)+\bE\left[\tau_1^C\right]
\end{equation}
The result follows after applying Proposition \ref{prop:low_noise_hinge_Etau1} with $\bm{\theta}_0=\bm{0}$.
  \end{proof}

 \begin{proof}[Proof of Theorem 3.1] Using Proposition \ref{prp:hinge_low_etm} and Lemma \ref{lem:stopping_time_appendix}, Theorem \ref{theorem_hinge}.1 immediately follows.
 \end{proof}

\subsubsection{High Variance Regime}\label{sec:high_hinge}
The main theorem of this section is as follows.
\begin{theorem}\label{thm:hinge_loss_high_noise}
Suppose that the step-size $\alpha$ satisfies
\begin{equation}
    \alpha< \frac{\min\{1,\tfrac{1}{2}r\rho^*\}}{\Vert \bm{\mu} \Vert^2+d\sigma^2}
\end{equation}
where $\rho^*$ is defined in Lemma \ref{lem:hinge_minimizer} and the constant $r$ is defined as
\begin{equation*}
    r:= \frac{64}{125 \rho^*\sigma \Vert \bm{\mu} \Vert \sqrt{2\pi}}\exp\left(-\frac{(\tfrac{4}{3}\frac{1}{\rho^*}-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2} \right).
\end{equation*}
Then it holds that $\bE[T]<+\infty$.
\end{theorem}
We refer the proof to the end of this section. We first need a technical lemma.
\begin{lemma}\label{lem:hinge_loss_high_noise_fundamental_lemma}
For any $\bm{\theta}$ the following holds
\begin{equation}\label{eq:lemma_for_high_noise_hinge}
    f(\bm{\theta}_{k-1})-f(\bm{\theta}) \leq \frac{1}{2\alpha} \left(\Vert \bm{\theta}_{k-1}-\bm{\theta}\Vert^2-\bE\left[\Vert \bm{\theta}_k-\bm{\theta}\Vert^2 |\mathcal{F}_{k-1}\right]\right)+\frac{\alpha}{2}\left(\Vert \bm{\mu} \Vert^2+d\sigma^2\right).
\end{equation}
\end{lemma}
\begin{proof}
See Appendix, Lemma \ref{lem:technical_convex_bound}. Note that we are using the inequality $\Vert \nabla_{\bm{\theta}}\ell(\bm{\xi}^T\bm{\theta})\Vert^2\leq \Vert \bm{\xi} \Vert^2$ and then applying \eqref{fact:norm_Gaussians}.
\end{proof}
We write the orthogonal complement of $\bm{\theta}\in \R^d$ as $\bm{\theta}=\rho\bm{\mu}+\tilde{\bm{\theta}}$. As before, the idea is to lower bound the left hand side of \eqref{eq:lemma_for_high_noise_hinge} for $\bm{\theta}=\bm{\theta}^*:=\rho^* \bm{\mu}$. To do so, we write $f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)=f(\bm{\theta}_{k-1})-f(\rho\bm{\mu})+f(\rho \bm{\mu})-f(\bm{\theta}^*)$ and then provide bounds for the both terms $f(\bm{\theta}_{k-1})-f(\rho_{k-1} \bm{\mu})$ and  $f(\rho_{k-1}\bm{\mu})-f(\rho^*\bm{\mu})$.
 The following lemma bounds $f(\bm{\theta}_{k-1})-f(\rho \bm{\mu})$.
\begin{lemma}\label{lem:hinge_high_noise_first_lemma} For each $\bm{\theta}\in \R^d$, the following bound holds
\begin{equation}
   f(\bm{\theta})-f(\rho\bm{\mu})\geq \max\left\{\tfrac{1}{\sqrt{2\pi}}\sigma \Vert \tilde{\bm{\theta}}\Vert-\rho \Vert \bm{\mu} \Vert\left(\sigma\sqrt{\tfrac{2}{\pi}}+\Vert \bm{\mu} \Vert \right)-1,0 \right\}.
\end{equation}
\end{lemma}
\begin{proof} We begin by denoting
$\xi_1:=\bm{\xi}^T\tilde{\bm{\theta}}$ and $\xi_2:=\bm{\xi}^T\bm{\mu}$. Notice that $\xi_1$ and $\xi_2$ are independent random variables. We have that
\begin{align*}
    f(\bm{\theta})-f(\rho \bm{\mu})&=\bE_{\bm{\xi}}\left[\ell(\bm{\xi}^T\bm{\theta})-\ell(\rho\bm{\xi}^T\bm{\mu}) \right]\\&=\bE_{\bm{\xi}}\left[\ell(\xi_1+\rho\xi_2)-\ell(\rho\xi_2) \right]\\&=\bE_{\bm{\xi}}\left[\ell(-\xi_1+\rho\xi_2)-\ell(\rho\xi_2) \right].
\end{align*}
The second equality follows since $\xi_1 \sim -\xi_1$. We define the function
\[
\Phi(\xi_1,\xi_2):=\ell(\xi_1+\rho\xi_2)+\ell(-\xi_1+\rho\xi_2)-2\ell(\rho\xi_2).
\]
We, therefore, obtain that
\[
2\left(f(\bm{\theta})-f(\rho\bm{\mu})\right)=\bE_{\xi_1,\xi_2}\left[\Phi(\xi_1,\xi_2)\right].
\]
We, next, claim that
\begin{equation}\label{eq:hinge_lemma_claim1}
    \Phi(\xi_1,\xi_2)=0 \text{ whenever } \vert \xi_1\vert\leq \vert 1-\rho\xi_2\vert.
\end{equation}
To see this, suppose that $\vert \xi_1\vert\leq \vert 1-\rho\xi_2\vert$ holds. We consider two cases.  First, assume that $0\leq 1-\rho\xi_2$. Then, it holds that $\rho\xi_2-\xi_1\leq 1$ and $\rho\xi_2+\xi_1\leq 1$. We then have $\Phi(\xi_1,\xi_2)=1-\xi_1-\rho\xi_2+1+\xi_1-\rho\xi_2-2(1-\rho\xi_2)=0$. Second, assume that $1-\rho\xi_2\leq 0$. Then, it holds that $1\leq \rho\xi_2-\xi_1$ and $1\leq \rho\xi_2+\xi_1$. Now it immediately follows that $\Phi(\xi_1,\xi_2)=0$. We have thus established \eqref{eq:hinge_lemma_claim1}. We, next, claim the following
\begin{equation}\label{eq:hinge_lemma_claim2}
    \Phi(\xi_1,\xi_2)=\vert \xi_1\vert-\vert 1-\rho\xi_2\vert \text{ whenever } \vert \xi_1\vert\geq \vert 1-\rho\xi_2\vert.
\end{equation}
To this end, we again consider two cases. First, assume that $\xi_1\leq -\vert 1-\rho\xi_2\vert$. It, then, holds that $1\leq -\xi_1+\rho\xi_2$ and $\xi_1+\rho\xi_2\leq 1$. We, therefore, have $ \Phi(\xi_1,\xi_2)= 1-\xi_1-\rho\xi_2-2\ell(\rho\xi_2)$. The claim \eqref{eq:hinge_lemma_claim2} follows from the following simple identity
\begin{equation}\label{eq:hinge_identity}
   2\ell(t)= 1-t+\vert 1 - t \vert, \quad \forall t \in \R.
\end{equation}
Second, assume that $\xi_1\geq \vert 1-\rho\xi_2\vert$. It, then, holds that $\xi_1+\rho\xi_2\geq 1$ and $-\xi_1+\rho\xi_2\leq 1$. We, therefore, have $
\Phi(\xi_1,\xi_2)=1+\xi_1-\rho\xi_2-2\ell(\rho \xi_2).
$ The claim \eqref{eq:hinge_lemma_claim2} follows from the identity \eqref{eq:hinge_identity}.  We, therefore, obtain
\begin{align}
    \bE_{\xi_1,\xi_2}[\Phi(\xi_1,\xi_2)]&=2\bE_{\xi_1,\xi_2}[(\ell(\xi_1+\rho\xi_2)+\ell(-\xi_1+\rho\xi_2)-2\ell(\rho\xi_2))1_{\{ \xi_1>0\}}]\label{eq:hinge_lemma_it1}\\&=2\bE_{\xi_1,\xi_2}[(\ell(\xi_1+\rho\xi_2)+\ell(-\xi_1+\rho\xi_2)-2\ell(\rho\xi_2))1_{\{ \xi_1\geq\vert 1-\rho \xi_2\vert\}}]\label{eq:hinge_lemma_it2}\\&=\bE_{\xi_1,\xi_2}[(\xi_1-\vert 1-\rho \xi_2\vert)1_{\{ \xi_1\geq \vert 1-\rho\xi_2\vert\}} ]. \label{eq:hinge_lemma_it3}
\end{align}
Here, \eqref{eq:hinge_lemma_it1} holds because $\xi_1\sim -\xi_1$ and $\Phi(\xi_1,\xi_2)=\Phi(-\xi_1,\xi_2)$. Equation \eqref{eq:hinge_lemma_it2} is true because of claim \eqref{eq:hinge_lemma_claim1} and \eqref{eq:hinge_lemma_it3} follows from claim \eqref{eq:hinge_lemma_claim2}. From \eqref{eq:hinge_lemma_it3}, we clearly conclude that $\bE_{\xi_1,\xi_2}[\Phi(\xi_1,\xi_2)]\geq 0$.  We, next, observe the bound
\begin{align}
    \bE_{\xi_1,\xi_2}[(\xi_1-\vert 1-\rho \xi_2\vert)1_{\{ \xi_1\geq \vert 1-\rho\xi_2\vert\}} ] &= \tfrac{1}{2}\bE_{\xi_1,\xi_2}[\xi_1-\vert 1-\rho \xi_2\vert+\vert \xi_1-\vert 1-\rho \xi_2\vert \vert]\nonumber\\&\geq -\tfrac{1}{2}\bE_{\xi_2}[\vert 1-\rho \xi_2\vert]+\tfrac{1}{2}\bE_{\xi_1,\xi_2}[\vert \xi_1\vert-\vert 1-\rho \xi_2\vert]\nonumber\\&=\tfrac{1}{2}\bE_{\xi_1}[\vert \xi_1\vert]-\bE_{\xi_2}[\vert1-\rho\xi_2\vert]. \label{eq:thrid_last_equation}
\end{align}
The second inequality follows from $\bE_{\xi_1}[\xi_1]=0$ and the triangle inequality $\vert x\vert-\vert y\vert \leq \vert \vert x\vert-y\vert$. On the other hand, it holds that
\begin{equation}\label{first_last_equation}
\bE_{\xi_1}[\vert \xi_1\vert]=\sqrt{\tfrac{2}{\pi}}\sigma \Vert \tilde{\bm{\theta}}\Vert,
\end{equation}
and
\begin{equation}\label{second_last_equation}
\bE_{\bm{\xi}}[\vert 1-\rho \bm{\mu}^T\bm{\xi}\vert]\leq 1+\rho \bE_{\bm{\xi}}[\vert \bm{\mu}^T\bm{\xi}\vert]\leq 1+\rho\Vert \bm{\mu} \Vert \left(\sigma \sqrt{\tfrac{2}{\pi}}+\Vert \bm{\mu} \Vert \right).
\end{equation}
Combing equations \eqref{eq:thrid_last_equation}
, \eqref{first_last_equation} and \eqref{second_last_equation} concludes the proof.
\end{proof}
In turn, we bound $f(\rho \bm{\mu})-f(\rho^*\bm{\mu})$ in the next lemma.
\begin{lemma}\label{lem:hinge_loss_second_lemma}
There exists a constant $r>0$ such that the following holds
\begin{equation}
    f(\rho \bm{\mu})-f(\rho^*\bm{\mu}) \geq r\vert \rho -\rho^*\vert \quad \text{for any $\rho$ satisfying $\vert \rho-\rho^*\vert >\tfrac{1}{2}\rho^*$ }.
\end{equation}
\end{lemma}
\begin{proof}
We begin by defining the function $h(\rho)=f(\rho\bm{\mu})$. Recall that
\[
f(\rho \bm{\mu})=\bE_{\bm{\xi} \sim \mathcal{P}_*}[\ell(\rho\bm{\xi}^T\bm{\mu})]=\bE_{\bm{\xi} \sim \mathcal{P}_*}[(1-\rho \bm{\xi}^T\bm{\mu})1_{\{\rho\bm{\xi}^T\bm{\mu}\leq 1 \}}]
\]
Hence, it holds that
\[
h'(\rho)=\bm{\mu}^T\nabla f(\rho \bm{\mu})=-\bE_{\bm{\xi} \sim \mathcal{P}_*}[\bm{\xi}^T\bmu1_{\{ \rho\bm{\xi}^T\bm{\mu}\leq 1\}}].
\]
From \eqref{eq:fact_affine}, we obtain that $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$.  For $\rho>0$, therefore, it holds that
\begin{equation}\label{der_phi_pos}
    h'(\rho)=\frac{-1}{\sigma\Vert \bm{\mu} \Vert\sqrt{2\pi}}\int_{-\infty}^{\frac{1}{\rho}}z\exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2)}{2\sigma^2\Vert \bm{\mu} \Vert^2} \right)dz.
\end{equation}
Applying chain rule thus yields
\[
h''(\rho)=\frac{1}{\rho^3\sigma\Vert \bm{\mu} \Vert\sqrt{2\pi}}\exp\left(-\frac{(\frac{1}{\rho}-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2}  \right) \quad \text{for all $\rho>0$}.
\]
Clearly, there exists $r>0$ such that for $\rho\in [\tfrac{3}{4}\rho^*,\tfrac{5}{4}\rho^*]$, it holds that $h''(\rho)\geq r$. Applying Lemma \ref{lem:convex_analysis_lemma} the result follows with the constant $r$ defined as follows
\begin{equation}\label{eq:constant_r_hinge_loss}
    r:= \frac{64}{125 \rho^*\sigma \Vert \bm{\mu} \Vert \sqrt{2\pi}}\exp\left(-\frac{(\tfrac{4}{3}\frac{1}{\rho^*}-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2} \right).
\end{equation}
\end{proof}
We define the target set $C$ as follows:
\begin{equation}\label{target_set_for_hinge}
    C:=\left\{\bm{\theta}:\vert \rho-\rho^*\vert \leq \tfrac{1}{2}\rho^* \text{ and } \Vert \tilde{\bm{\theta}}\Vert \leq c:=\frac{\sqrt{2\pi}}{\sigma}\left(2+\tfrac{3}{2}\rho^*\Vert \bm{\mu} \Vert\left(\sigma\sqrt{\tfrac{2}{\pi}}+\Vert \bm{\mu} \Vert \right) \right) \right\}.
\end{equation}
Also, we consider the following drift function
\begin{equation}
    V(\bm{\theta}):=\Vert \bm{\theta} -\bm{\theta}^*\Vert^2.
\end{equation}
The following proposition establishes the drift equation.
\begin{proposition}
Suppose that the step-size $\alpha$ satisfies the bound
\begin{equation}\label{bound_for_alpha_hinge_loss}
    \alpha< \frac{\min\{1,\tfrac{1}{2}r\rho^*\}}{\Vert \bm{\mu} \Vert^2+d\sigma^2}
\end{equation}
where $r$ is defined in \eqref{eq:constant_r_hinge_loss}. The following is then true.
\begin{equation}
    \left(\bE\left[V(\bm{\theta}_k)|\mathcal{F}_{k-1} \right]-V(\bm{\theta}_{k-1})\right)1_{\{\bm{\theta}_{k-1}\not\in C\}}\leq -\tfrac{1}{2}\min\{1,\tfrac{1}{2}r\rho^*\}1_{\{\bm{\theta}_{k-1}\not\in C\}}.
\end{equation}
\end{proposition}

\begin{proof}
Suppose that $\bm{\theta}_{k-1}\not\in C$.  Lemma \ref{lem:hinge_loss_second_lemma} yields that
\begin{equation}\label{eq:hinge_lemma_proposition_eq2}
     f(\rho_{k-1}\bm{\mu})-f(\rho^*\bm{\mu}) \geq \tfrac{1}{2}r\rho^* \quad \text{ whenever } \vert \rho-\rho^* \vert \geq \tfrac{1}{2}\rho^*
\end{equation}
where constant $r$ is defined in \eqref{eq:constant_r_hinge_loss}. Moreover, from Lemma \ref{lem:hinge_high_noise_first_lemma}, we obtain that
\begin{equation}\label{eq:hinge_lemma_proposition_eq1}
    f(\bm{\theta}_{k-1})-f(\rho_{k-1} \bm{\mu})\geq 1 \quad \text{ whenever } \sigma \Vert \tilde{\bm{\theta}}_{k-1}\Vert \geq c \text{ and } \vert \rho-\rho^* \vert < \tfrac{1}{2}\rho^*.
\end{equation}
Moreover,
Combining \eqref{eq:hinge_lemma_proposition_eq1},   \eqref{eq:hinge_lemma_proposition_eq2} yields that
\begin{equation}
    f(\bm{\theta}_{k-1})-f(\rho^*\bm{\mu}) \geq \min\{1,\tfrac{1}{2}r\rho^*\} \quad \text{ whenever } \bm{\theta}_{k-1} \not\in C.
\end{equation}
Lemma \eqref{lem:hinge_loss_high_noise_fundamental_lemma} and the bound on $\alpha$ in \eqref{bound_for_alpha_hinge_loss} along with the above displayed bound gives that
\begin{equation}
    \bE\left[V(\bm{\theta}_k)|\mathcal{F}_{k-1} \right]-V(\bm{\theta}_{k-1}) \leq -\tfrac{1}{2}\min\{1,\tfrac{1}{2}r\rho^*\}.
\end{equation}
The proof is complete.
\end{proof}
\begin{proof}[Proof of Theorem \ref{thm:hinge_loss_high_noise}]
The target set $C$ defined in \eqref{target_set_for_hinge} is compact and thus there exists $b>0$ such that
\begin{equation}
    \left( \bE\left[V(\bm{\theta}_k)|\mathcal{F}_{k-1} \right]-V(\bm{\theta}_{k-1})\right)1_{\{\bm{\theta}_{k-1}\in C\}} \leq b1_{\{\bm{\theta}_{k-1}\in C\}}.
\end{equation}
Therefore, it holds that
\begin{equation}\label{drift_for_hinge_loss}
    \bE\left[V(\bm{\theta}_k)|\mathcal{F}_{k-1} \right]-V(\bm{\theta}_{k-1}) \leq -\tfrac{1}{2}\min\{1,\tfrac{1}{2}r\rho^*\}1_{\{\bm{\theta}_{k-1}\not\in C\}}+b1_{\{\bm{\theta}_{k-1}\in C\}}.
\end{equation}
Furthermore, since $C$ is a compact set which does not contain $\bm{0}$, there exists a positive constant $\delta$ such that for any fixed $\bm{\theta}\in C$ it holds that
\begin{equation}
    \delta:= \min \mathbb{P}_{\bm{\xi} \sim \mathcal{P}_*}\left(\bm{\xi}^T\bm{\theta}\geq 1 \right)>0.
\end{equation}
The result now follows from Appendix Lemma \ref{lem:drift_lemma_appendix} and Lemma \ref{lem:stopping_time_appendix}.
\end{proof}

\section{Introduction}
Minimization of an expected loss objective function using linear predictors,
\begin{equation} \label{eq:intro_expected_loss} \min_{\bm{\theta} \in \R^d} f(\bm{\theta}) := \bE_{(\bm{\zeta},y) \sim \mathcal{P}} \ell(\bm{\zeta}^T\bm{\theta}, y), \end{equation}
is a central task in machine learning. Here the loss function $\ell: \R \times \R \to \R$, the probability distribution $\mathcal{P}$ is unknown, and the data sample $(\bm{\zeta}, y) \in \R^d \times \R$ is a random vector distributed as $\mathcal{P}$. The most prevalent algorithm employed for solving \eqref{eq:intro_expected_loss} is \textit{stochastic gradient descent} (SGD). Whereas a significant amount of work has been devoted to the convergence analysis of SGD (see, {\em e.g.}, \cite{RM1951,Curtis_SGD,Bubeck_Convex_book, Pflug}), leading, in particular, to learning rate schedules, the question of how to terminate the algorithm when one is near an optimal classifier remains largely unaddressed.

 Yet, inexpensive stopping criteria are of utmost interest in machine learning. For instance, if one could produce a low cost test to determine near-optimality, then without sacrificing the quality of the solution or efficiency of the SGD algorithm, needless computational time would be eliminated. Secondly, early termination tests impose a degree of predictability on accuracy and running times-- a useful quality when SGD occurs as a subproblem of a larger computation. Several works show that early termination of SGD can prevent overfitting, speed up learning procedures, and/or improve generalization properties \citep{Prechelt2012, Hardt:2016:TFG:3045390.3045520,Yao2007}. Motivated by these facts, we sought to address from stochastic optimization the following question:
\begin{center}
    How to design a test to terminate SGD with a fixed learning rate that is inexpensive without sacrificing quality of the solution?
\end{center}
To do so, we simplified our setting to binary classification, one of the fundamental examples of supervised machine learning  \citep{ShalevShwartzBenDavid}. In binary classification, the learning algorithm is given a sequence of training examples $(\bm{\zeta}_1,y_1),(\bm{\zeta}_2,y_2),\ldots$, often noisy, where $\bm{\zeta}_i\in\R^d$ and $y_i\in\{0,1\}$ for each $i$.  The job of the algorithm is to develop a rule for distinguishing future, unseen $\bm{\zeta}$'s that are classified as $1$ from those classified as $0$. In this work, we limit attention to linear classifiers.  This means that the learning algorithm must determine a vector $\bm{\theta}$ such that the classification of $\bm{\zeta}$ is $1$ when $\bm{\zeta}^T\bm{\theta}>0$ else it is $0$. Note that any algorithm for linear classification can be extended to one for nonlinear classification via the construction of ``kernels''; see, {\em e.g.}, \cite{ShalevShwartzBenDavid}. This extension is not pursued; we leave it for later work.

The usual technique for determining $\bm{\theta}$, which is also adopted herein, is to define a loss function that turns the discrete problem of computing a $1$ or $0$ for $\bm{\zeta}$ to a continuous quantity. Common choices of loss functions include logistic and hinge. For simplicity, we consider only the unregularized logistic and hinge loss in this work.

 Our theoretical results assume that our data comes from a Gaussian mixture model (GMM). The GMM is attributed to \cite{article}.  The problem of identifying GMM parameters given random samples has attracted considerable attention in the literature; see, \textit{e.g}., the recent work of \cite{Ashtiani} and earlier references therein. Another common
use of GMMs in the literature, similar to our application here, is as test-cases for a learning algorithm intended to solve a more general problem. Examples include clustering; see, \textit{e.g.,} \cite{DBLP:journals/corr/abs-1902-07137} and \cite{pmlr-v70-panahi17a} and tensor factorization; see, \textit{e.g.,} \cite{sherman2019estimating}.

Ordinarily in deterministic first-order optimization methods, one terminates when the norm of the gradient falls below a predefined tolerance.  In the case of SGD for binary classification, this is unsuitable for two reasons. First, the true gradient is generally inaccessible to the algorithm or it is computationally expensive to generate even a sufficient approximation of the gradient.

 Second, even if the computations were possible, an `optimal' classifier $\bm{\theta}$ for the classification task is not necessarily the minimizer of the loss function since the loss function is merely a surrogate for correct classification of the data.

\paragraph{Our contributions.} In this paper, we introduce a new and simple termination criterion for stochastic gradient descent (SGD) applied to binary classification using logistic regression and hinge loss with constant step-size $\alpha>0$. Notably, our proposed criterion adds no additional computational cost to the SGD algorithm.

We analyze the behavior of the classifier at termination, where we sample from a normal distribution with unknown means $\bm{\mu}_0,\bm{\mu}_1\in \R^d$ and variances $\sigma^2I_d$. Here  $\sigma>0$ and $I_d$ is the $d \times d$ identity matrix. As such, we make no assumptions on the separability of the data set.

When the variance is not too large, we have the following results:
\begin{enumerate}
    \item The test will be activated for any fixed positive step-size. In particular, we establish an upper bound for the expected number of iterations before the activation occurs. This upper bound tends to a numeric constant  when $\sigma$  converges to zero. In fact, we show that the expected time until termination decreases linearly as the data becomes more separable ({\em i.e.}, as the noise $\sigma \to 0$).
    \item  We prove that the accuracy of the classifier at termination nearly matches the accuracy of an optimal classifier. Accuracy is the fraction of predictions that a classification model got right while an optimal classifier minimizes the probability of misclassification when the sample is drawn from the same distribution as the training data.
\end{enumerate}
When the variance is large, we show that the test will be activated for a sufficiently small step-size.

We empirically evaluate the performance of our stopping criterion versus a baseline competitor. We compare performances on both synthetic (Gaussian and heavy-tailed $t$-distribution) as well as real data sets (MNIST \citep{MNIST} and CIFAR-10 \citep{cifar10}). In our experiments, we observe that our test yields relatively accurate classifiers with small variation across multiple runs.

\paragraph{Related works.}  To the best of our knowledge, the earliest comprehensive numerical testing of a stopping termination test for SGD in neural networks was introduced by \cite{Prechelt2012}.
His stopping criteria, which we denote as \textit{small validation set} (SVS), periodically checks the iterate on a validation set.  Theoretical guarantees for SVS were established in the works of \citep{Early_stopping_Lin,Yao2007}. \cite{Hardt:2016:TFG:3045390.3045520} shows that SGD is uniformly stable and thus solutions with low training error found quickly generalize well. These results support exploring new computationally inexpensive termination tests-- the spirit of this paper.

In a related topic,
the relationship between generalization and optimization is an active area of research in machine learning. Much of the pioneering work in this area focused on understanding how early termination of algorithms, such as conjugate gradient, gradient descent, and SGD, can act as an implicit regularizer and thus exhibit better generalization properties \citep{Prechelt2012,Early_stopping_Lin,Yao2007,CG_implicit_regularization,Multipass_Lin}.  The use of early stopping as a tool for improving generalization is not studied herein because our experiments indicate that for the problem under consideration, binary classification with a linear separator, the accuracy increases as SGD proceeds and ultimately reaches a steady value but does not decrease, meaning that there is no opportunity to improve generalization by stopping early.
See also \cite{Nemirovski_Robust_Stochastic_1}.

Instead of using a validation set to stop early, \cite{Variational_Duvenaud} employs an estimate of the marginal likelihood as a stopping criteria. Another termination test based upon a Wald-type statistic developed for solving least squares with reproducing kernels guarantees a minimax optimal testing \citep{Liu_stopping}. However it is unclear the practical benefits of such procedures over a validation set.

Several works have introduced validation procedures to check the accuracy of solutions generated from stochastic algorithms based upon finding a point $\bm{\theta}_\varepsilon$ that satisfies a high confidence bound $\bP(f(\bm{\theta}_\varepsilon) - \min f \le \varepsilon) \ge 1-p$, in essence, using this as a stopping criteria (\textit{e.g.}, see \cite{Dima_Robust_Stochastic,Ghadimi_Strongly_cvx_validation_2,Ghadimi_Strongly_cvx_validation_1, Juditsky_robust_stochastic,Nemirovski_Robust_Stochastic_1}).
Yet, notably, all these procedures produce points with small function values. For binary classification, however, this could be quite expensive and a good classifier need not necessarily be the minimizer of the loss function. Ideally, one should terminate when the classifier's direction aligns with the optimal direction-- the approach we pursue herein.

\subsection{Low regime, proof of Theorem \ref{thm:low}}\label{sec:proof_low}
In this section, we investigate the low variance regime. We consider the target set $C$ and function $V$ defined in \eqref{eq:targetset_lownoise} and \eqref{eq:driftfunction_lownoise} respectively, \textit{i.e.}
\begin{equation}\label{eq:CV_low_recall}
      C = \{\bm{\theta}: \bm{\mu}^T\bm{\theta}\geq 1 \}, \quad V(\bm{\theta})=\left(M-\bm{\mu}^T\bm{\theta}\right)^2,
\end{equation}
where $M$ is a constant to be determined.
Next lemma shows that the drift equation \eqref{eq:driftequation} holds for the pair $(C,V)$.  \begin{lemma}[Drift equation]\label{lem:driftequation} Consider the SGD algorithm and let the set $C$ and the function $V$ be as in \eqref{eq:CV_low_recall}. Define the constants $c,b,M$ as in \eqref{eq:para_log} and \eqref{eq:para_hinge}. Then provided that $\sigma \leq c\Vert \bm{\mu} \Vert$, the function $V$ is a drift function with respect to the set $C$ and it satisfies the drift equation \eqref{eq:driftequation} with the constant $b$.
\end{lemma}
\begin{proof}For simplicity we write $\mathcal{F}_{-1}:=\sigma\left(\{\bm{\theta}_0=\bm{\theta}\} \right)$. Fix $k\geq 1$ and write $\bm{\xi}_k=\bm{\mu}+\sigma\bm{\psi}_k$ with $\bm{\psi}_k \sim N(0,I_d)$. Denote $\psi_k:=\frac{\bm{\mu}^T\bm{\psi}_k}{\Vert\bm{\mu} \Vert}$, thus $\psi_k\sim N(0,1)$. In order to show that the function $V$ satisfies the drift equation \eqref{eq:driftequation}, it suffices to assume $\bm{\theta}_{k-1}\not\in C$; in particular, this means $\bm{\theta}_{k-1}^T\bm{\mu}<1$.
\item
\textbf{Logistic loss.} By expanding out the term using the update formula, we get the following
\begin{align} \label{eq: low_noise_blah_1}
    V(\bm{\theta}_{k})= V(\bm{\theta}_{k-1})-\frac{2\alpha \bm{\mu}^T\bm{\xi}_k(M-\bm{\mu}^T\bm{\theta}_{k-1})}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}+\frac{\alpha^2(\bm{\mu}^T\bm{\xi}_k)^2}{(1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1}))^2}.
\end{align}
We have
\begin{align*}
 &\bE_{\bm{\xi}_k}\left[ \frac{\bm{\mu}^T\bm{\xi}_k}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]\\
 & \qquad =\Vert\bm{\mu}\Vert^2\bE_{\bm{\xi}_k}\left[ \frac{1}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]+\sigma \Vert \bm{\mu}\Vert\bE_{\bm{\xi}_k,\psi_k}\left[ \frac{\psi_k}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]\\
 &  \qquad \geq \Vert \bm{\mu} \Vert^2 \bE_{\bm{\xi}_k}\left[ \frac{1}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]+\sigma \Vert \bm{\mu}\Vert\bE_{\psi_k}\left[\psi_k1_{\{\psi_k<0\}} \right]\\
  &  \qquad = \Vert \bm{\mu} \Vert^2 \bE_{\bm{\xi}_k}\left[ \frac{1}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})} \left ( 1_{\{\bm{\mu}^T \bm{\theta}_{k-1} \geq  \bm{\xi}_k^T \bm{\theta}_{k-1}\}} + 1_{\{\bm{\mu}^T \bm{\theta}_{k-1} < \bm{\xi}_k^T \bm{\theta}_{k-1}\}} \right ) |\mathcal{F}_{k-1}\right] -\sigma \Vert \bm{\mu}\Vert \sqrt{\frac{1}{2\pi}}\\
  &  \qquad \geq \frac{\Vert \bm{\mu}\Vert^2}{1+\exp(\bm{\mu}^T\bm{\theta}_{k-1})}\bE_{\bm{\xi}_k}\left[ 1_{\{\bm{\mu}^T \bm{\theta}_{k-1} \geq  \bm{\xi}_k^T \bm{\theta}_{k-1}\}} |\mathcal{F}_{k-1} \right]-\sigma \Vert \bm{\mu}\Vert \sqrt{\frac{1}{2\pi}}\\
 &\qquad \geq \frac{\Vert \bm{\mu}\Vert^2}{2(1+e)}-\sigma \Vert \bm{\mu}\Vert \sqrt{\frac{1}{2\pi}}\\
 & \qquad \geq 0.001\Vert \bm{\mu} \Vert^2 .
\end{align*}
Here the first inequality follows from $\bE[X] \ge \bE[X1_{\{X<0\}}]$ and $1+ \exp(\bm{\xi}_k^T\bm{\theta}_{k-1}) \ge 1$, the second equation from \eqref{fact:norm_Gaussians}, and the second to last from the observation that for any $X$ normally distributed, $\bP(\bE[X] \ge X) = 1/2$ and $\bm{\xi}_k^T\bm{\theta}_{k-1} \sim N(\bm{\mu}^T\bm{\theta}_{k-1}, \sigma^2 \norm{\bm{\theta}_{k-1}}^2)$ and $\bm{\mu}^T\bm{\theta}_{k-1} < 1$. The last inequality uses the assumption $\sigma \le 0.33 \norm{\bm{\mu}}$. By taking the conditional expectations of \eqref{eq: low_noise_blah_1} combined with the above sequence of inequalities, we deduce the following bound
\begin{align*}
    &\bE\left[V(\bm{\theta}_{k})-V(\bm{\theta}_{k-1})|\mathcal{F}_{k-1}\right]\\
    & \quad \qquad = \bE_{\bm{\xi}_k} \left [-\frac{2\alpha \bm{\mu}^T\bm{\xi}_k(M-\bm{\mu}^T\bm{\theta}_{k-1})}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})} | \mathcal{F}_{k-1} \right]  + \bE_{\bm{\xi}_k} \left [\frac{\alpha^2(\bm{\mu}^T\bm{\xi}_k)^2}{(1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1}))^2} | \mathcal{F}_{k-1} \right]\\
    & \quad \qquad \leq -0.002(M-1)\alpha \Vert \bm{\mu} \Vert^2+ \alpha^2\Vert \bm{\mu} \Vert^2\left(\Vert \bm{\mu}\Vert^2+\sigma^2 \right)\\
    & \quad \qquad = \alpha \Vert \bm{\mu} \Vert^2\left[ -0.002(M-1)+\alpha\left(\Vert \bm{\mu} \Vert^2+\sigma^2\right)\right].
\end{align*}
Here the first inequality follows from $\bm{\mu}^T\bm{\theta}_{k-1} < 1$ and by upper bounding $\frac{(\bm{\mu}^T\bm{\xi}_k)^2}{(1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1}))^2}$ with $(\bm{\mu}^T\bm{\xi}_k)^2$ and then applying \eqref{fact:norm_Gaussians}.
A quick computation after plugging in the value of $M$ and the bound $\sigma\leq 0.33\Vert \bm{\mu} \Vert$ from \eqref{eq:para_log} yields the drift equation \eqref{eq:driftequation} with $b=\alpha \Vert \bm{\mu} \Vert^2$.
\item \textbf{Hinge loss.} By expanding out the term using the update formula, we get the following
\begin{equation}\label{eq:Vequation_hinge}
   V(\bm{\theta}_k)=V(\bm{\theta}_{k-1})-2\alpha (M-\bm{\mu}^T\bm{\theta}_{k-1})\bm{\mu}^T\bm{\xi}_k1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}+\alpha^2(\bm{\mu}^T\bm{\xi}_k)^21_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}.
\end{equation}
We have
\begin{align*}
    \bE_{\bm{\xi}_k}[1_{\{ \bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\bm{\mu}^T\bm{\xi}_k|\mathcal{F}_{k-1}] &=\Vert \bm{\mu}\Vert^2\bE_{\bm{\xi}_k}[1_{\{ \bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}|\mathcal{F}_{k-1}]+\sigma\Vert \bm{\mu} \Vert\bE_{\bm{\xi}_k,\psi_k}[1_{\{ \bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}\psi_k|\mathcal{F}_{k-1}]\\
    & \geq \frac{1}{2}\Vert \bm{\mu} \Vert^2+\sigma\Vert \bm{\mu} \Vert\bE_{\psi_k}[\psi_k1_{\{\psi_k<0 \}}]\\
    & = \frac{1}{2}\Vert \bm{\mu} \Vert^2-\sigma\Vert \bm{\mu} \Vert\sqrt{\frac{1}{2\pi}} \\&
    \geq
    0.001 \Vert \bm{\mu} \Vert^2.
\end{align*}
Here the first inequality follows from $1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq \bm{\mu}^T\bm{\theta}_{k-1}\}}\leq 1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}$ and  $\bE_{\bm{\xi}_k}[1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq \bm{\mu}^T\bm{\theta}_{k-1}\}}]=\tfrac{1}{2}$, and the second from \eqref{fact:norm_Gaussians}. The last inequality uses the assumption $\sigma \leq 1.25 \Vert \bm{\mu} \Vert$. By taking conditional expectations of \eqref{eq:Vequation_hinge} combined with the above sequence of inequalities, we deduce the bound
\begin{equation*}
\begin{aligned}
\bE[V(\bm{\theta}_k)-V(\bm{\theta}_{k-1})|\mathcal{F}_{k-1}]&=\bE_{\bm{\xi}_k}\left[-2\alpha(M-\bm{\mu}^T\bm{\theta}_{k-1})1_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}|\mathcal{F}_{k-1}\right]+\bE_{\bm{\xi}_k}\left[\alpha^2(\bm{\mu}^T\bm{\xi}_k)^21_{\{\bm{\xi}_k^T\bm{\theta}_{k-1}\leq 1\}}|\mathcal{F}_{k-1}\right]
\\
&\leq \alpha \Vert \bm{\mu} \Vert^2\left[ -0.002(M-1)+\alpha\left(\Vert \bm{\mu} \Vert^2+\sigma^2\right)\right].
\end{aligned}
\end{equation*}
A quick computation after plugging in the value of $M$ and the bound $\sigma \leq 1.25 \Vert \bm{\mu} \Vert$ yields the desired result.
\end{proof}
 Recall, the stopping times $\tau_m$ denote the $m^{th}$ time that the SGD iterates enter the target set $C$. We show that $\bE[\tau_m]=\mathcal{O}(m)$. To do so, we begin by stating a lemma that gives a bound on the stopping time $\tilde{\tau}_1$ starting from any $\bm{\theta}_0$. In other words, for an arbitrary starting $\bm{\theta}_0$, we define
 \[
 \tilde{\tau}_1:=\inf\{k>0:\bm{\theta}_k\in C\}.
 \]
\begin{lemma}[\cite{meyn2012markov}, Theorem 11.3.4]\label{lem:drift_from_meyn} Suppose that $V:\R^d\rightarrow [0,+\infty)$ is a drift function with respect to some target set $C$ \textit{i.e.} for some constant $b\in (0,+\infty)$ the drift equation \eqref{eq:driftequation} holds. The following is true
\begin{equation}
    \bE[\tilde{\tau}_1|\bm{\theta}_0=\bm{\theta}]\leq \tfrac{1}{b}V(\bm{\theta}).
\end{equation}
\end{lemma}
We establish upper bounds on $\bE[\tau_m]$ for $m\geq 1$ in the following proposition.
\begin{proposition}(Bound on $\bE[\tau_m]$)\label{prp:Etaum_lownoise}
Let $\bm{\theta}_0=\bm{0}$ and assume the notation and assumptions of Lemma \ref{lem:driftequation} hold. The following is true for all $m\geq 1$
\begin{align}
    \bE[\tau_m]\leq (m-1) \left ( 1+\frac{M^2}{b}\cdot\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\frac{\alpha\sigma^3 M^2}{ \Vert \bm{\mu} \Vert b}\cdot\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right) \right ) +\frac{M^2}{b}.
\end{align}
\end{proposition}
\begin{proof}First, the result for $m=1$ follows immediately by combining Lemmas \ref{lem:driftequation} and \ref{lem:drift_from_meyn} with $\bm{\theta}_0 = \bm{0}$. We now assume that $\tau_{m-1}<\infty$ a.s. for some $m\geq 2$. Fix an integer $n \ge 1$. We decompose the space to yield the following bounds

 \begin{equation} \begin{aligned} \label{eq: low_noise_bound}
    \bE\big[(\tau_m-\tau_{m-1})\wedge& n|\mathcal{F}_{\tau_{m-1}+1}\big]=\bE\left[((\tau_m-\tau_{m-1})\wedge n)|\mathcal{F}_{\tau_{m-1}+1}\right]1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\geq 1\}}\\
    & \qquad \qquad \qquad \qquad  \qquad +\bE\left[((\tau_m-\tau_{m-1})\wedge n)|\mathcal{F}_{\tau_{m-1}+1}\right]1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}<1\}}\\
    &= 1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\geq 1\}}+ \bE\left[((\tau_m-\tau_{m-1})\wedge n)|\mathcal{F}_{\tau_{m-1}+1}\right]1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}<1\}}\\
    &=1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\geq 1\}}+\sum_{i=1}^{\infty}\bE\left[(\tau_m-\tau_{m-1})\wedge n|\mathcal{F}_{\tau_{m-1}+1}\right]1_{\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\leq i\}}\\
&= 1+\sum_{i=1}^{\infty}\bE\left[\tilde{\tau}_1\wedge n|\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}+1}\right]1_{\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\leq i\}}.
\end{aligned} \end{equation}
Here the first equality follows because $((\tau_m-\tau_{m-1})\wedge n) 1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1} \ge 1\}} = 1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\geq 1\}}$ and the last equality by the strong Markov property.  We consider the logistic and hinge loss case separately to show that the following is true
\begin{equation}\label{eq:ineq:indicators}
\begin{aligned}
1_{\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\leq i\}} &\leq 1_{\{\bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}<\frac{1-i}{\alpha}\}}.
\end{aligned}
\end{equation}
For clarity, in the next few inequalities, we write $1\{.\}$ instead of $1_{\{.\}}$. In case of logistic loss,
for each $i \ge 1$, we observe the bound
\begin{equation}
\begin{aligned}
 1\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\leq i\}&\leq 1\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\}\nonumber\\&=1\left\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}}-\frac{\alpha \bm{\mu}^T\bm{\xi}_{{\tau_{m-1}}+1}}{1+\exp(\bm{\xi}_{{\tau_{m-1}}+1}^T\bm{\theta}_{\tau_{m-1}})}\right\}\nonumber\\
    &\leq 1\left\{i-1<-\frac{\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}}{1+\exp(\bm{\xi}_{\tau_{m-1}+1}^T\bm{\theta}_{\tau_{m-1}})}\right\} \label{eq:i-1_calculus}\nonumber\\
    & \leq 1\left\{i-1<-\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}\right\},\nonumber
\end{aligned}
\end{equation}
where the second inequality follows because $\bm{\mu}^T\bm{\theta}_{\tau_{m-1}} \geq 1$ and the last inequality because $-\alpha \bm{\mu}^T \bm{\xi}_{\tau_{m-1}+1}$ is positive since $i-1\geq 0$.
 \item In case of hinge loss, for each $i\geq 1$, similar as above, we observe the bound
 \begin{equation}
     \begin{aligned}
     1\left\{i-1<1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\leq i\right\}&\leq 1\left\{i-1<1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\right\}\\&\leq 1\left\{i-1<1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}}-\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}1_{\{\bm{\xi}_{\tau_{m-1}+1}^T\bm{\theta}_{\tau_{m-1}}\leq 1\}}\right\}\\&\leq 1\left\{i-1<-\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}1_{\{\bm{\xi}_{\tau_{m-1}+1}^T\bm{\theta}_{\tau_{m-1}}\leq 1\}} \right\}\\&= 1\left\{i-1<-\alpha\bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}\right\}.
     \end{aligned}
 \end{equation}
 Therefore we have shown that \eqref{eq:ineq:indicators} holds. Setting $\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}+1}$ , by Lemma \ref{lem:drift_from_meyn} for each $i\geq 1$, we deduce
\begin{equation} \begin{aligned}  \label{eq: low_noise_blah_3}
\bE\left[\tilde{\tau}_1\wedge n|\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}+1}\right]1_{\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\leq i\}}&\leq  \frac{(M-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1})^2}{b}1_{\{i-1< 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}+1}\leq i\}}\\&\leq  \frac{(M+i-1)^2}{b}1_{\{ \bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}< \frac{1-i}{\alpha}\}}.
\end{aligned} \end{equation}
Finally we observe that
\begin{equation} \begin{aligned} \label{eq: low_noise_blah_4}
    \bE\left[1_{\{ \bm{\mu}^T\bm{\xi}_{\tau_{m-1}+1}< \frac{1-i}{\alpha}\}} \right]&=\bE\left[\sum_{k=1}^{\infty}1_{\{ \bm{\mu}^T\bm{\xi}_{k+1}< \frac{1-i}{\alpha}\}}1_{\{\tau_{m-1}=k\}}\right]\\
&=\sum_{k=1}^{\infty}\bE\left[1_{\{ \bm{\mu}^T\bm{\xi}_{k+1}< \frac{1-i}{\alpha}\}}  \right]\bE\left[1_{\{\tau_{m-1}=k\}} \right]\\
    &= \Phi\left(\frac{\frac{1-i}{\alpha}-\Vert \bm{\mu}\Vert^2}{\sigma\Vert \bm{\mu}\Vert } \right)\sum_{k=1}^{\infty}\bE\left[1_{\{\tau_{m-1}=k\}} \right]\\
    &=\Phi\left(\frac{\frac{1-i}{\alpha}-\Vert \bm{\mu}\Vert^2}{\sigma\Vert \bm{\mu}\Vert } \right).
\end{aligned} \end{equation}
The second equality is by independence and the third equality because $\bm{\mu}^T \bm{\xi}_{k+1} \sim N(\norm{\bm{\mu}}^2, \sigma^2 \norm{\bm{\mu}}^2)$.  By combining  \eqref{eq: low_noise_bound}, \eqref{eq: low_noise_blah_3}, and \eqref{eq: low_noise_blah_4}, we obtain the following
\begin{equation} \begin{aligned} \label{eq: low_noise_blah_5}
    \bE\big[(\tau_m-&\tau_{m-1})\wedge n\big]\leq 1+\frac{M^2}{b}\cdot\Phi\left(-\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\sum_{i=2}^{\infty} \frac{(M+i-1)^2}{b}\cdot\Phi\left(\frac{\frac{1-i}{\alpha}-\Vert \bm{\mu}\Vert^2}{\sigma\Vert \bm{\mu}\Vert } \right)\\
    &=1+\frac{M^2}{b}\cdot\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\sum_{i=2}^{\infty} \frac{(M+i-1)^2}{b}\cdot\Phi^{c}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert } \right)\\
    &\le1+\frac{M^2}{b}\cdot\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\frac{\alpha\sigma\Vert \bm{\mu} \Vert}{b\sqrt{2\pi}}\cdot\sum_{i=2}^{\infty} \frac{(M+i-1)^2}{\alpha\Vert \bm{\mu}\Vert^2+i-1}\cdot\exp\left(-\frac{1}{2}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert}\right)^2 \right),
\end{aligned} \end{equation}
where we used the inequality $\Phi^{c}(t)<\frac{1}{t\sqrt{2\pi}}\exp(-\frac{t^2}{2})$ for all $t>0$. Next, note that $\frac{M+i-1}{\alpha \Vert \bm{\mu} \Vert^2+i-1}\leq \frac{M}{\alpha \Vert\bm{\mu}\Vert^2}$ holds for all $i\geq 2$.  Using this we obtain the following bound
\begin{equation} \begin{aligned}\label{eq:exp_low_noise}
       \sum_{i=2}^{\infty} \frac{(M+i-1)^2}{\alpha\Vert \bm{\mu}\Vert^2+i-1}\cdot\exp&\left(-\frac{1}{2}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert}\right)^2 \right)\\
       &\leq \frac{\sigma M^2}{\alpha \Vert \bm{\mu} \Vert^3}\cdot\sum_{i=2}^{\infty} \frac{\alpha\Vert \bm{\mu}\Vert^2+i-1}{\alpha \sigma \Vert \bm{\mu} \Vert}\cdot\exp\left(-\frac{1}{2}\left(\frac{\alpha\Vert \bm{\mu}\Vert^2+i-1}{\alpha\sigma\Vert \bm{\mu}\Vert}\right)^2 \right)\\&\leq \frac{\sigma M^2}{\alpha \Vert \bm{\mu} \Vert^3}\cdot \alpha\sigma \Vert \bm{\mu} \Vert\cdot\int_{\frac{\Vert \bm{\mu} \Vert}{\sigma}}^{+\infty} t\exp\left(-\frac{t^2}{2}\right)dt \\&=\frac{\sigma^2 M^2}{\Vert \bm{\mu} \Vert^2}\cdot\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right).
\end{aligned} \end{equation}
Here we have used that $t \mapsto t\exp(-\frac{t^2}{2})$ is decreasing over $[1,+\infty)$. Combining \eqref{eq: low_noise_blah_5} and \eqref{eq:exp_low_noise}, we obtain that
\begin{equation}\begin{aligned}
 \bE\left[(\tau_m-\tau_{m-1})\wedge n\right]&\leq 1+\frac{M^2}{b}\cdot\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\frac{\alpha\sigma^3 M^2}{ \Vert \bm{\mu} \Vert b}\cdot\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right).
\end{aligned}
\end{equation}
Taking the limit as $n \to +\infty$, we observe that
\[\bE[\tau_m]\leq 1+\frac{M^2}{b}\cdot\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\frac{\alpha\sigma^3 M^2}{ \Vert \bm{\mu} \Vert b}\cdot\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right)+\bE[\tau_{m-1}].
\]
We then iterate the above inequality yielding
\[ \bE[\tau_m] \le (m-1) \left ( 1+\frac{M^2}{b}\cdot\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\frac{\alpha\sigma^3 M^2}{ \Vert \bm{\mu} \Vert b}\cdot\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right)\right ) + \bE[\tau_1]. \]
The result follows by plugging in the bound from Lemma \ref{lem:drift_from_meyn} for the base case $m=1$.
\end{proof}
We are now ready to prove Theorem~\ref{thm:low}.
\begin{proof}[Proof of Theorem
\ref{thm:low}]
In order to simplify the subsequent argument, we define the quantity,
\[ M' := 1+\frac{M^2}{b}\cdot\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma} \right)+\frac{\alpha\sigma^3 M^2}{ \Vert \bm{\mu} \Vert b}\cdot\frac{1}{\sqrt{2\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right).  \]
It is easy to see that $\mathbb{P}_{\hat{\bm{\xi}}\sim N(\bm{\mu},\sigma^2I_d)}\left( \hat{\bm{\xi}}^T\bm{\theta}\geq 1 \right) \geq \frac{1}{2}$ for any $\bm{\theta}\in C$. Therefore $\delta=\frac{1}{2}$ satisfies \eqref{eq:probability_delta}. By Proposition~\ref{prp:Etaum_lownoise} with Lemma \ref{lem:ETleqET_C}, we conclude that
\begin{align*}
\bE[T]&\leq \bE[T_C]=\sum_{m=1}^{\infty} \bE[T_C1_{\{T_C=\tau_m\}}]\leq \sum_{m=1}^{\infty}\frac{\bE[\tau_m]}{2^{m-1}} \le \sum_{m=1}^\infty \frac{(m-1)M' + \frac{M^2}{b} }{2^{m-1}} = 2M' + \frac{2M^2}{b}.
\end{align*}
\end{proof}

\section{Numerical Experiments} \label{sec:Num_Experiment}

We investigate the performance of our termination test on two popular data sets, MNIST \citep{MNIST} and CIFAR-10 \citep{cifar10}, as well as synthetic data generated from Gaussians and heavy-tailed student t-distributions. All tests were performed using our zero overhead stopping criteria outlined in \eqref{eq: practical_termination_test}; experiments using our test which required an extra sample \eqref{eq: termination_test} are not presented since the behaviors of the two criteria were indistinguishable on all data sets.

\paragraph{Comparison with a popular stopping criterion.} We include as a baseline a popular termination test, the small validation set (SVS) \citep{Prechelt2012}. The SVS termination test is as follows. One fixes a validation set of $p$ instances $(\bm{\zeta}^{\rm V}_1,y^{\rm V}_1)$, \ldots, $(\bm{\zeta}^{\rm V}_p,y^{\rm V}_p)$ drawn from the same distribution as the training data. Then for $m = 1, 2, \ldots$, one checks the fraction correct of the current classifier $\bm{\theta}_{ml}$, where $ml$ is the iteration index, on the $p$ instances.  In other words, the SVS test is run once every $l$ iterations.   If the fraction correct fails to increase compared to the last run of the SVS, then the SGD iterations are terminated.

Note the computational overhead of running the small validation set is about $p$ times the cost of one SGD iteration.  Therefore, in order to make the overhead only a constant factor, we choose $l=2p$, meaning an approximately 50\% overhead for SVS. In contrast, the overhead for \eqref{eq: practical_termination_test} is
0. The value of $p$ is a tuning parameter for SVS; we exhibit results for three different $p$ values (see Figs.~\ref{fig:normal_scatter}, \ref{fig:ht_scatter}, \ref{fig:mnist_scatter}, \ref{fig:cifar_scatter} ).

\begin{figure*}[htp!]
\begin{center}
\subfigure[]{\includegraphics[height=3cm]{it_vs_sigma_l.eps}} \quad
\subfigure[\label{fig:black}]{\includegraphics[height=3cm]{acc_vs_sigma_l.eps}} \quad
\subfigure[]{\includegraphics[height=3cm]{it_vs_sigma_h.eps}} \quad \subfigure[]{\includegraphics[height=3cm]{acc_vs_sigma_h.eps}}
\end{center}
\caption{Performance of stopping criterion \eqref{eq: practical_termination_test} on a mixture of Gaussians as $\sigma$ is varied.  Plots $(a),(b)$ are logistic and $(c),(d)$ are hinge.  All plots show tests for values of $\sigma$ equally spaced from 0.05 to 2.0.  For each value of $\sigma$, ten trials were run. Plots $(a),(c)$ show the relationship between $\sigma$ and $k$, the iteration number when \eqref{eq: practical_termination_test} first holds. Plots $(b),(d)$ show the accuracy as red asterisks.  The green asterisks show the accuracy of the optimal classifier.  The black curve on the right is the ratio of the average accuracy (over 10 trials) of the classifier
when \eqref{eq: practical_termination_test} holds to the accuracy of the optimal classifier.}
\label{fig:acctime}
\end{figure*}

\paragraph{Measuring the accuracy.} In all the experiments, we measure the performance of a method with a score, generally known as ``accuracy," that is the fraction correct on a large validation set drawn from the same distribution as the training data.  Thus, 1.0 is perfect accuracy, while 0.5 means that $\bm{\theta}_k$ is no better at classifying than random guessing.  It is important to note that even on data for which the means $\bm{\mu}_0,\bm{\mu}_1$ are known a priori ({\em e.g.}, synthetic data), the score of the optimal $\bm{\theta}^*$
will not be 1.0 because the large validation set itself is noisy.

We center the data so that the linear classifier is homogeneous. In a preliminary phase, 100 samples are drawn from the training set. From this, $\bm{\mu}_0$ and $\bm{\mu}_1$ are estimated, and then the average of these estimates is used to offset training instances during SGD.

\paragraph{Parameter settings.} After centering, the vectors $\bm{\theta}$ and $\bm{\xi}$ scale inversely, so the step-size parameter $\alpha$ should scale as $1/\sigma^2$.  Therefore, we take the step-size to be
$\tilde\alpha/\tilde\sigma^2$.  Here, $\tilde\sigma^2$ is the average of $\norm{\bm{\zeta}_j-\tilde\bm{\mu}_{y_j}}^2$, and $\tilde\bm{\mu}_i$ ($i=0$ or $i=1$)  is the estimate of $\bm{\mu}_i$, averaged over the two classes. We compute the quantities $\tilde\sigma^2$ and $\tilde\bm{\mu}_i$ using the 100 samples described in the preceding paragraph. Note that for the Gaussian mixture model, the expected value of $\tilde\sigma^2$ is $\sigma^2d$.
For the synthetic data, the means and variances are known exactly a priori, so the estimation procedures described in the previous two paragraphs are unnecessary.  However, we used them anyway in order to be consistent with the tests on the realistic data.

The parameter $\tilde\alpha$ described in the last paragraph is a scale-free tuning parameter.
It is known (see, e.g., \cite{Nemirovski_Robust_Stochastic_1}) that a smaller $\tilde\alpha$ corresponds to more iterations but greater ultimate accuracy under a reasonable model of the data.  Our termination test is obviously sensitive to the choice of $\tilde\alpha$: the condition $\bm{\xi}_{k+1}^T\bm{\theta}_k\ge 1$ cannot hold unless $\norm{\bm{\theta}_k}\ge 1/\norm{\bm{\xi}_{k+1}}$, but $\bE\left[\norm{\bm{\theta}_k}\right]\le O(\alpha k)$. See also Theorems~\ref{thm:low} and \ref{thm:high}.
On the other hand, SVS is only mildly sensitive to $\tilde\alpha$, according to our testing.  Indeed, there is an upper bound of $pl$ on the total number of iterations possible before termination using the SVS condition, independent of $\tilde\alpha$ and of all other aspects of the problem.  The dependence of the termination test on $\tilde\alpha$ is evidently desirable because the user is presumably seeking greater accuracy when a smaller value of $\tilde\alpha$ is selected.

\subsection{Experiments with synthetic data}
\label{sec:comp-sim}

\begin{figure*}[htp!]
\centering
  \includegraphics[height=2.5cm]{gm_scatter99_500_l_10.eps} \qquad \includegraphics[height=2.5cm]{gm_scatter99_500_l_200.eps} \qquad
\includegraphics[height=2.5cm]{gm_scatter99_500_h_10.eps}\qquad \includegraphics[height=2.5cm]{gm_scatter99_500_h_200.eps} \\ \vspace{2 mm}
\includegraphics[height=2.5cm]{gm_scatter100_500_l_10.eps}\qquad \includegraphics[height=2.5cm]{gm_scatter100_500_l_200.eps} \qquad
\includegraphics[height=2.5cm]{gm_scatter100_500_h_10.eps}\qquad \includegraphics[height=2.5cm]{gm_scatter100_500_h_200.eps} \\ \vspace{2 mm}
 \includegraphics[height=2.5cm]{gm_scatter101_500_l_10.eps}\qquad \includegraphics[height=2.5cm]{gm_scatter101_500_l_200.eps} \qquad    \includegraphics[height=2.5cm]{gm_scatter101_500_h_10.eps}\qquad \includegraphics[height=2.5cm]{gm_scatter101_500_h_200.eps}
 \caption{Each plot shows 10 random runs of SGD applied to normally distributed data with indicated values of $\sigma$ and for a fixed dimension $d=500$. For each of the ten runs, five termination tests corresponding to five colors were applied.  SVS was tried with $p=32, 128, 512$, depicted as red, magenta and cyan circles respectively.  Test \eqref{eq: practical_termination_test} is indicated with a blue asterisk. A green `+' corresponds to termination after $1.5k$ iterations, where $k$ is the iteration index that \eqref{eq: practical_termination_test} first holds.  The notation $(l/200)$ means logistic loss with $\tilde\alpha=1/200$; simillarly $(h/10)$ means hinge loss with $\tilde\alpha = 1/10$, and so on.}
\label{fig:normal_scatter}
\end{figure*}

\paragraph{Normal distribution.} We generated test and training data using a mixture of Gaussians given by $N(\bz,\sigma^2I)$ for the 0-class and $N(\bm{e}_1,\sigma^2I)$ for the 1-class, where $\bm{e}_1 = (1,0,\hdots, 0)^T \in \R^d$.

In Fig.~\ref{fig:acctime}, we present the running time and accuracy (fraction correct) of our termination test for a fixed dimension $d=500$ and $\sigma$ ranging from $0.05$ to $2$. We record 10 runs for each value of $\sigma$. The performance of the classifier when our termination test \eqref{eq: practical_termination_test} holds almost matches the optimal classifier; in particular, the averaged accuracy of our classifier/accuracy of the optimal classifier over the 10 runs, black curve in Fig.~\ref{fig:black}, never dips below $0.95$.

In Fig.~\ref{fig:normal_scatter}, we compare performance of \eqref{eq: practical_termination_test} against SVS termination. One axis shows accuracy while the other shows iteration count. We continued to run SGD for an additional $1.5k$ iterations where $k$ is the first iteration at which \eqref{eq: practical_termination_test} holds (green '+') to test whether accuracy improves after termination.
The tests (for several values of $\sigma$, both hinge and logistic, and two values of $\tilde\alpha$) in Fig.~\ref{fig:normal_scatter} indicate that \eqref{eq: practical_termination_test} is more accurate than SVS, more predictable (i.e., there is less spread in the scatter plot), and that running until $1.5k$ iterations does not significantly improve the solution.  As expected,
for a large $\tilde\alpha$, \eqref{eq: practical_termination_test} requires fewer iterations than SVS with $p=512$, while the opposite relationship holds for a small $\tilde\alpha.$

\begin{figure}
    \begin{center}
    \includegraphics[height=2.5cm]{ht_scatter98_2_l_10.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter98_2_l_200.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter98_2_h_10.eps} \qquad
 \includegraphics[height=2.5cm]{ht_scatter98_2_h_200.eps} \qquad       \includegraphics[height=2.5cm]{ht_scatter99_2_l_10.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter99_2_l_200.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter99_2_h_10.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter99_2_h_200.eps} \\ \vspace{2 mm}
    \includegraphics[height=2.5cm]{ht_scatter100_2_l_10.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter100_2_l_200.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter100_2_h_10.eps} \qquad \includegraphics[height=2.5cm]{ht_scatter100_2_h_200.eps} \\
    \end{center}
        \caption{Tests on the student-t distribution (heavy tailed) with two degrees of freedom and the indicated value of parameter $\beta$.  See the caption of Fig.~\ref{fig:normal_scatter} for explanation of the plots.}
    \label{fig:ht_scatter}
\end{figure}

\paragraph{Heavy-tailed distribution.}
We consider the student t-distribution with two degrees of freedom.  This
distribution is heavy-tailed since some of its
higher moments are infinite.

The two classes were generated as follows.  For $\bm{\zeta}$ in the 0-class, each of the $d$ entries of $\bm{\zeta}$ is chosen as
$\beta\eta$, where $\beta$ is varied in the experiments and
$\eta$ is drawn from the student t-distribution with two degrees of freedom.  For the 1-class, $\bm{\zeta}$ is chosen in the same way except that the first entry is incremented by 1. Fig.~\ref{fig:ht_scatter} shows our performance against SVS.
The results in this table show similar trends as in the normally distributed case.  One difference is that the accuracy achieved by our termination test \eqref{eq: practical_termination_test} is more spread out presumably because of the heavy-tailed nature of the data set.

\subsection{Experiments with real data} \label{sec:comp-realistic}

\paragraph{MNIST handwritten digits.} We compared our termination test on the MNIST handwritten digit set \citep{MNIST} ($d=784$, no preprocessing of the data other than centering between the two means).  Two trials are shown: distinguishing 1 from 8 (easy case) and distinguishing 7 from 9 (more difficult case).  The test runs are obtained by running through the training data in different randomized orders.  The plots in Fig.~\ref{fig:mnist_scatter} show similar trends as before.  As expected, the accuracy is overall higher for
$\tilde\alpha=1/200$ than for $\tilde\alpha=1/10$.

\begin{figure}
    \centering
    \includegraphics[height=2.5cm]{mnist_scatter1_8_l_10.eps} \qquad     \includegraphics[height=2.5cm]{mnist_scatter1_8_l_200.eps} \qquad  \includegraphics[height=2.5cm]{mnist_scatter1_8_h_10.eps} \qquad     \includegraphics[height=2.5cm]{mnist_scatter1_8_h_200.eps} \\  \vspace{2 mm} \includegraphics[height=2.5cm]{mnist_scatter7_9_l_10.eps} \qquad     \includegraphics[height=2.5cm]{mnist_scatter7_9_l_200.eps} \qquad \includegraphics[height=2.5cm]{mnist_scatter7_9_h_10.eps} \qquad     \includegraphics[height=2.5cm]{mnist_scatter7_9_h_200.eps} \\
    \caption{Tests on the MNIST handwritten digit data set for discerning ``1'' from ``8'' and ``7'' from ``9'' for both hinge and logistic, and for both $\tilde\alpha=1/10$ and $\tilde\alpha=1/200$.  Refer to the caption of Fig.~\ref{fig:normal_scatter} for the key to the plots.}
    \label{fig:mnist_scatter}
\end{figure}

\paragraph{CIFAR-10 image set.} We compared our termination test on the CIFAR-10 \citep{cifar10} ($d=3072$, no preprocessing of the data other than centering between the two means as described earlier).  Two trials are shown: distinguishing deer from airplanes and frogs from trucks. As in MNIST, test runs are obtained by running through the training data in different randomized orders.

\begin{figure}
    \centering
    \includegraphics[height=2.5cm]{cifar_scatter_1_5_l_10.eps} \qquad \includegraphics[height=2.5cm]{cifar_scatter_1_5_l_200.eps} \qquad \includegraphics[height=2.5cm]{cifar_scatter_1_5_h_10.eps} \qquad \includegraphics[height=2.5cm]{cifar_scatter_1_5_h_200.eps} \\ \vspace{2 mm}
\includegraphics[height=2.5cm]{cifar_scatter_7_10_l_10} \qquad
 \includegraphics[height=2.5cm]{cifar_scatter_7_10_l_200.eps} \qquad \includegraphics[height=2.5cm]{cifar_scatter_7_10_h_10.eps} \qquad \includegraphics[height=2.5cm]{cifar_scatter_7_10_h_200.eps}
        \caption{Tests on the CIFAR-10 image set for two tasks, for logistic and hinge losses, and for $\tilde\alpha=1/10$ and $\tilde\alpha=1/200$.   Refer to the caption of Fig.~\ref{fig:normal_scatter} for the key to the plots.  The plot in the first row, right, does not include cyan circles because the training data was exhausted before the SVS test could activate for $p=512$.}
    \label{fig:cifar_scatter}
\end{figure}

\section{Conclusions}
We have proposed a simple and computationally free termination test for SGD for binary classification, supported by both theoretical and experimental results.
The theoretical results show that the test will stop SGD after a finite time with a bound on the expected accuracy of the resulting classifier.  The bounds that we proved are weaker than what we observed in our experiments.  Therefore, the first obvious question left open by this work is whether the theoretical bounds can be improved.

In our experimental results,
the plots in Figs.~\ref{fig:normal_scatter} through \ref{fig:cifar_scatter} show a consistent pattern that
\eqref{eq: practical_termination_test} achieves low accuracy but is faster than SVS for $\tilde\alpha=1/10$, while it achieves higher accuracy with more iterations when $\tilde\alpha=1/200$.  This is useful behavior in practice, compared to SVS, since it puts the accuracy/iterations tradeoff in the hands of the user who selects the stepsize $\tilde\alpha$. Another benefit of \eqref{eq: practical_termination_test} apparent from all plots is that the number of iterations is more consistent across random trials, which is beneficial in the case that SGD is used as a subproblem of a larger computation.

This work did not explore regularization via early stopping.  As mentioned in the introduction, experiments showed that as SGD iterations continued, the accuracy on the test set eventually levels off but does not decrease significantly, {\em i.e.}, SGD for binary classification is not prone to overfitting.  Because the test accuracy never shows marked decline, there is no opportunity for early stopping to regularize.  However, we know of other settings in which early stopping has a strong regularizing effect ({\em e.g.}, conjugate gradient iterations for image deconvolution, already known in \cite{CG_implicit_regularization}), so if \eqref{eq: practical_termination_test} is extended beyond binary classification in future work, there will likely also be an opportunity to explore regularization.

\section{Stopping criterion for stochastic gradient descent}\label{sec:termtest} We analyze learning by minimizing an expected loss problem of homogeneous linear predictors ({\em i.e.}, without bias) of the form
\begin{align*}
    \bE_{(\bm{\zeta},y) \sim \mathcal{P}} [\ell(\bm{\zeta}^T\bm{\theta}, y)]
\end{align*}
using logistic and hinge regression. Here the samples $(\bm{\zeta}, y) \in \R^d \times \{0,1\}$. We recall that in logistic regression the loss function is defined as follows
\begin{equation}\label{eq:log_loss_definition}
    \ell(x,y):=-yx+\log\left(1+\exp(x)\right).
\end{equation}
Also, the hinge loss is defined as the following
\begin{equation}\label{eq:hinge_loss_definition}
    \begin{aligned}
    \ell(x,y):=\begin{cases} \max(1-x,0)& \quad y=1, \\\max(1+x,0)& \quad y=0. \end{cases}
    \end{aligned}
\end{equation}

 The data comes from a mixture model, that is, flip a coin to determine whether an item is in the $y=0$ or $y=1$ class, then generate the sample $\bm{\zeta}$ from either the distribution $\mathcal{P}_0$ (if $y=0$ was selected) or $\mathcal{P}_1$ (if $y=1$ was selected). We denote the mean of the $\mathcal{P}_0$ (resp. $\mathcal{P}_1$) distribution by $\bm{\mu}_0$ (resp. $\bm{\mu}_1$).
The homogeneity of the linear classifier is without loss of much generality because we can assume $\bm{\mu}_0 = - \bm{\mu}_1$. We enforce this assumption, with minimal loss in accuracy, by recentering the data using a preliminary round of sampling (see Sec.~\ref{sec:Num_Experiment}).

Because of the homogeneity, we can
simplify the notation by redefining our training examples to be $\bm{\xi}_k:=(2y_k-1)\bm{\zeta}_k$ and then assuming that for all $k \ge 0$, $y_k = 1$.  Then the new samples $\bm{\xi}$ can be drawn from a \textit{single}, mixed distribution $\mathcal{P}_*$ with mean $\bm{\mu}:=\bm{\mu}_1$ where sampling $\bm{\xi}\sim \mathcal{P}_1$ occurs with probability 0.5 and $-\bm{\xi}\sim\mathcal{P}_0$ occurs with probability 0.5.
We make this simplification and, from this point on, we analyze the following optimization problem:
\begin{equation}\label{optimization_problem}
\min_{\bm{\theta} \in \R^d} f(\bm{\theta}):= \bE_{\bm{\xi} \sim \mathcal{P}_*}[ \ell(\bm{\xi}^T\bm{\theta},1)]
\end{equation}
Let us remark that the right-hand side of \eqref{optimization_problem} is differentiable with respect to $\bm{\theta}$ in either cases of logistic and hinge loss functions. Indeed, in case of hinge loss, note that for any $\bm{\theta}_{k-1}$, the function $\bm{\xi}_k \mapsto \ell(\bm{\xi}_k^T\bm{\theta}_{k-1},1)$ is almost surely differentiable as $\bP_{\bm{\xi}_k}\left(\bm{\xi}_k^T\bm{\theta}_{k-1}=1\right)=0$. Hence, we consider the expectation in \eqref{optimization_problem} to be over $\mathbb{R}^d\backslash \left\{\bm{\xi}_k:\bm{\xi}_k^T\bm{\theta}_{k-1}=1 \right\}$ on which the argument is differentiable with respect to $\bm{\theta}_{k-1}$.

The most widely used method to solve \eqref{optimization_problem} is SGD. Unlike gradient descent which uses the entire data to compute the gradient of the objective function, the SGD algorithm, at each iteration, generates a sample from the probability distribution and updates the iterate based only on this sample,
\begin{align}
 \label{eq:derivative_formula}
 \bm{\te}_{k}=\bm{\theta}_{k-1} -\alpha \nabla_{\bm{\theta}} \ell(\bm{\xi}_{k}^T\bm{\theta}_{k-1},1),
\end{align}
where $\bm{\xi}_k \sim \mathcal{P}_*$.  Our presentation of SGD assumes a constant step-size $\alpha>0$. Constant step-size is commonly used in machine learning implementations despite the decreasing step-size often assumed to prove convergence (see, \textit{e.g.}, \cite{RM1951}). \cite{Nemirovski_Robust_Stochastic_1} explain in more detail the theoretical basis for both constant and decreasing step-size and provide an explanation as well as workarounds for the poor practical performance of decreasing step-size.  However, in practice, constant step-size is still widely used.
With constant step-size, SGD is known to asymptotically converge to a neighborhood of the minimizer (see, {\em e.g.}, \cite{Pflug}). Yet, for binary classification, one does not require convergence to a minimizer in order to obtain good classifiers.

For homogeneous linear classifiers applied to the hinge loss function, it has been shown (\cite{molitor2019bias}) that the homotopic sub-gradient method converges to a maximal margin solution on linearly separable data. In (\cite{Srebro_logistic}), SGD applied to the logistic loss on linearly separable data will produce a sequence of $\bm{\theta}_k$ that diverge to infinity, but when normalized also converge to the $L_2$-max margin solution. Little is known about the behavior of constant step-size SGD when the linear separability assumption on the data is removed (see, \textit{e.g.}, \citep{Telgarsky_logistic}). The assumption of zero-noise in our context would mean that $\mathcal{P}_0$, $\mathcal{P}_1$ each reduce to a single point, a trivial example of separable data. Since there is often noise in the sample procedure, the data \textit{may not necessarily be linearly separable}. Understanding the behavior of SGD in the presence of noise is, therefore, important.

\subsection{Stopping criterion} A common stopping criterion from deterministic first-order optimization methods is to terminate at an iterate satisfying $\norm{\nabla f(\bm{\theta})}^2 < \varepsilon$ for a predetermined $\varepsilon > 0$. Yet, in stochastic optimization, the full gradient is inaccessible or it is simply too expensive to compute. Several works \citep{Dima_Robust_Stochastic,Ghadimi_Strongly_cvx_validation_2,Ghadimi_Strongly_cvx_validation_1, Juditsky_robust_stochastic,Nemirovski_Robust_Stochastic_1} have suggested an alternative for the stochastic setting-- terminate when $\bP(f(\bm{\theta})-\min~f \le \varepsilon) \ge 1-p$ for some chosen small $\varepsilon >0$ and probability $p$.
However, for binary classification, the minimizer of the loss function and a perfect classifier may not be the same or one may find a suitable substitute, at a lower cost, without having to compute the exact minimizer.

\paragraph{Optimal classifiers.} In classification, we call a classifier, $\bm{\theta}^*$, \textit{optimal} if it has the property that
\begin{equation} \label{eq:stop_criteria_observation}
\bm{\theta}^*\in \Argmax_{\bm{\theta}} \bP\left(\bm{\xi}^T\bm{\theta}>0\,|\,\bm{\xi}\sim \mathcal{P}_*\right), \end{equation}
\textit{i.e.}, the classifier, $\bm{\theta}^*$, minimizes the probability of misclassifying. Note there exist many optimal classifiers, in fact, the condition \eqref{eq:stop_criteria_observation} is scale-invariant; hence, for any $\lambda > 0$, $\lambda \cdot \bm{\xi}^T \bm{\theta}^* > 0 \Longleftrightarrow \bm{\xi}^T \bm{\theta}^* > 0$.
Even though the binary classifier is scale-free, the logistic and hinge regression loss is not.  It transitions from flat to unit-slope when $\bm{\xi}^T\bm{\theta}=O(1)$. This suggests that when  $\bm{\theta}$ reaches this region, a classification has been made.

\paragraph{Termination test.} Motivated by the above property of optimal classifiers, we propose the following termination test: Sample $\hat{\bm{\xi}}_k \sim \mathcal{P}_*$ and
\begin{align} \label{eq: termination_test}
  \text{Terminate when \;$ \hat{\bm{\xi}}_k^T\bm{\theta}_k \ge 1$}.
\end{align}
A second motivation for this termination test comes from support vector machine (SVM) theory
\citep{ShalevShwartzBenDavid} in which the scaling of the optimizing classifier is constrained so that the margin between classes is $O(1)$.
Therefore, our termination test blends an SVM notion with SGD. Algorithm~\ref{alg:SGD_termination} describes the termination criteria \eqref{eq: termination_test} as applied with the update rule governed by SGD.

The termination test \eqref{eq: termination_test} requires an additional sample and an additional inner product per iteration and, as such,  imposes a small additional cost. To reduce this cost, in all our numerical experiments (Sec.~\ref{sec:Num_Experiment}), we use the following termination test.
\begin{equation}\label{eq: practical_termination_test}
\mbox{Terminate when \;$ \bm{\xi}_{k+1}^T\bm{\theta}_k\ge 1,$}
\end{equation}
which imposes no computational overhead as SGD already computes $\bm{\xi}_{k+1}^T\bm{\theta}_k$. Unfortunately, we could not perform a straightforward analysis of \eqref{eq: practical_termination_test} because it introduces additional dependencies in the sequences $\{\bm{\xi}_k\}_{k=1}^\infty$ and $\{\bm{\theta}_k\}_{k=0}^\infty$.
After testing both \eqref{eq: termination_test} and \eqref{eq: practical_termination_test}, we found that up to the noise from the randomness, their behaviors in numerical experiments were identical.

\begin{algorithm}[htp!]
\textbf{initialize:} $\bm{\theta}_0 \in \R^d$, $\alpha > 0$, $\hat{\bm{\xi}}_0 \sim \mathcal{P}_*$, $k = 0$\\
\textbf{while $\hat{\bm{\xi}}_k^T \bm{\theta}_k < 1$}\\
\qquad \qquad Pick data point $\bm{\xi}_{k+1} \sim \mathcal{P}_*$.\\
\qquad \qquad Compute $\nabla_{\bm{\theta}} \ell(\bm{\xi_{k+1}}^T\bm{\theta}_k,1)$ as in \eqref{eq:derivative_formula} \\
\qquad \qquad Update $\bm{\theta}$ by setting
\begin{equation}\label{eq: SGD_update_expected_loss} \bm{\theta}_{k+1} \leftarrow \bm{\theta}_k - \alpha \nabla_{\bm{\theta}} \ell(\bm{\xi_{k+1}}^T\bm{\theta}_k,1) \end{equation}\\
\qquad \qquad Sample $\hat{\bm{\xi}}_{k+1} \sim \mathcal{P}_*$\\
\qquad \qquad $k \leftarrow k+1$\\
\textbf{end}
\caption{SGD with termination test} \label{alg:SGD_termination}
\end{algorithm}

\begin{assumption}\label{Gaussian_Assumption}[The distribution $\mathcal{P}_{*}$ is Gaussian] \rm{Our theoretical analysis makes a further assumption on the distribution $\mathcal{P}_*$. For the rest of this section and Sec.~\ref{sec:Analysis}, $\mathcal{P}_0=N(\bm{\mu}_0,\sigma^2I_d)$, $\mathcal{P}_1=N(\bm{\mu}_1,\sigma^2I_d)$, and therefore $\mathcal{P}_*=N(\bm{\mu},\sigma^2I_d)$, a Gaussian with unknown mean $\bm{\mu}\; (=\bm{\mu}_1=-\bm{\mu}_0)$ and variance $\sigma^2I_d$. This assumption allows for non-separable data provided $\sigma > 0$.}
\end{assumption}

\paragraph{The minimizer of logistic and hinge regression}
In \eqref{eq:stop_criteria_observation} we defined $\bm{\theta}^*$ to be any member of the set of optimal classifiers.  For the remainder of this section, we provide an exact characterization of this set.  In the next lemma, we redefine $\bm{\theta}^*$ to the minimizer of the expected loss function for either hinge or logistic and show that it is a positive scalar multiple of $\bm{\mu}$.  We will continue to use $\bm{\theta}^*$ with this meaning for the remainder of the paper.  In the lemma after that, we show that the set of optimal classifiers are exactly positive scalar multiples of $\bm{\mu}$ (or of $\bm{\theta}^*$).

\begin{lemma}[Minimizer of the logistic and hinge loss] \label{lem:minimizers} The function $f$ defined in \eqref{optimization_problem} with $\ell$ defined in \eqref{eq:log_loss_definition} or \eqref{eq:hinge_loss_definition} has a unique minimizer at $\bm{\theta}^* = \rho^*\bm{\mu}$ for some $\rho^*\in (0,+\infty)$. Moreover, let $r=\rho^*\sigma^2$. Then in the case of logistic regression, it holds that $r=2$ and in the case of hinge loss, $w=\frac{\sigma}{r\Vert \bm{\mu} \Vert}-\frac{\Vert \bm{\mu} \Vert}{\sigma}$ satisfies
\begin{equation}\label{eq:w_hinge}
\frac{1}{\sqrt{2\pi}}\cdot\frac{\sigma}{\Vert \bm{\mu} \Vert}=\Phi(w)\cdot \exp(\tfrac{1}{2}w^2).
\end{equation}
\end{lemma}

\begin{proof}
We consider the logistic and hinge loss case separately.
\item \textbf{Logistic loss.} We have
\[
f(\bm{\theta})=\bE_{\bm{\xi} \sim N(\bm{\mu},\sigma^2I_d)}[ -\bm{\theta}^T\bm{\xi} + \log(1+ \exp(\bm{\theta}^T\bm{\xi}))].
\]
Clearly, $f$ is a convex function. We next observe that for any $\bm{v},\bm{\theta}\in \R^d$ with $\bm{v}^T\bm{\theta}=0$, it holds that
\begin{equation}\label{app.eq:min_eq101}
  \bm{v}^T\nabla f\left(\bm{\theta}\right)=\bE_{\bm{\xi}}\left[\frac{\bm{\xi}^T\bm{v}}{1+\exp(\bm{\xi}^T\bm{\theta})}\right]=\bE_{\bm{\xi}}[\bm{\xi}^T\bm{v}]\bE_{\bm{\xi}}\left[\frac{1}{1+\exp(\bm{\xi}^T\bm{\theta})}\right]=\bm{v}^T\bm{\mu}\cdot\bE_{\bm{\xi}}\left[\frac{1}{1+\exp(\bm{\xi}^T\bm{\theta})}\right].
\end{equation}
Here we used that $\bm{\xi}^T\bm{v}$ and $\bm{\xi}^T\bm{\theta}$ are independent random variables and the expectation of the product of two uncorrelated random variables is the product of the expectations.  Now note that for any $\bm{\theta}$, the quantity  $\bE_{\bm{\xi}}\left[\frac{1}{1+\exp(\bm{\xi}^T\bm{\theta})}\right]$ is strictly positive. Therefore, if  $\bm{v}^T\bm{\theta}=0$ and $\nabla f(\bm{\theta})=\bm{0}$ then, using \eqref{app.eq:min_eq101}, we obtain that $\bm{v}^T\bm{\mu}=0$. Hence, we established that $\nabla f(\bm{\theta})=\bm{0}$ implies $\bm{\theta}=\rho \bm{\mu}$ for some $\rho \in \R$.
On the other hand, using \eqref{app.eq:min_eq101} again, we have that $\nabla f(\rho \bm{\mu})=0$ if and only if $\bm{\mu}^T\nabla f(\rho \bm{\mu})=0$. To see the only if direction, suppose $\bm{\mu}^T\nabla f(\rho \bm{\mu}) = 0$ and $\nabla f(\rho \bm{\mu}) \neq 0$. Then we have $\nabla f(\rho \bm{\mu}) = \bm{v}$ where the vector $\bm{v}$ is nonzero such that $\bm{v}^T\bm{\mu} = 0$. By \eqref{app.eq:min_eq101}, we deduce $\norm{\bm{v}}^2 = \bm{v}^T \nabla f(\rho \bm{\mu}) = 0$ yielding a contradiction.

Next, we consider the function,
\begin{equation*}
  g(\rho):=-\bE_{\bm{\xi}}\left[\frac{\bm{\mu}^T\bm{\xi}}{1+\exp(\rho{\bm{\mu}}^T\bm{\xi})} \right].
\end{equation*}
Observe that $g(\rho) = \bm{\mu}^T \nabla f(\rho \bm{\mu})$. Therefore, if we can show $g(\rho)$ has a unique zero at $\rho = \tfrac{2}{\sigma^2} =: \rho^*$, we can conclude that $\bm{\mu}^T\nabla f(\rho^* \bm{\mu}) = 0$ which, in turn, gives us that $\rho^* \bm{\mu}$ is the unique solution to $\nabla f(\rho^* \bm{\mu}) = 0$. It remains to show that $\rho^*$ is the unique zero of $g$. By \eqref{eq:fact_affine}, $z: = \bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu}\Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$. Therefore, this yields
\begin{equation*}
    g(\rho)=\frac{1}{\sigma\Vert \bm{\mu} \Vert \sqrt{2\pi}} \int_{-\infty}^{\infty} \frac{z}{1+\exp(\rho z)} \exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2} \right)dz.
\end{equation*}
Expanding out the term inside the integral, we conclude
\begin{align}
    \frac{z}{1+\exp(\rho z)} \exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2} \right)&=\frac{z}{2\cosh\left(\frac{\rho z}{2}\right)}\exp\left(-\frac{\rho z}{2}-\frac{(z-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2} \right)\nonumber\\
    &=\frac{z}{2\cosh\left(\frac{\rho z}{2}\right)}\exp\left(-\frac{z^2+\left(\rho\sigma^2\Vert \bm{\mu} \Vert^2-2\Vert \bm{\mu} \Vert^2 \right)z+\Vert \bm{\mu} \Vert^4}{2\sigma^2\Vert \bm{\mu} \label{app.eq:integrho^*}
    \Vert^2}\right).
  \end{align}
When $\rho = \rho^*$, we observe that equation \eqref{app.eq:integrho^*} is an odd function of $z$. Therefore, the function $g(\rho^*) = 0$, i.e. the integral of \eqref{app.eq:integrho^*} is $0$. To see that $\rho^*$ is the only zero of $g$, we note that
\[
g'(\rho)=\bE_{\bm{\xi}}\left[\frac{\left(\bm{\mu}^T\bm{\xi} \right)^2\exp(\rho \bm{\mu}^T\bm{\xi})}{(1+\exp(\rho \bm{\mu}^T\bm{\xi}))^2} \right] >0.
\]
Here, $g'(\rho)=0$ implies that $\bm{\mu}^T\bm{\xi}=0$ a.s. which is not true. As a result, the function $g(\rho)$ is strictly decreasing with a zero at $\rho^*$. The result follows.
\item \textbf{Hinge loss.} We begin by noting that $f$ is differentiable and it holds that
\[
\nabla f(\bm{\theta})= -\bE_{\bm{\xi}}[\bxi1_{\{\bm{\xi}^T\bm{\theta}\leq 1\}}].
\]
We next observe that for any $\bm{v},\bm{\theta}\in \R^d$ such that $\bm{v}^T\bm{\theta}=0$, it holds that
\begin{equation}\label{eq:hinge_min_lemma}
    -\bm{v}^T\nabla f(\bm{\theta})=\bE_{\bm{\xi}}[\bm{v}^T\bxi1_{\{\bm{\xi}^T\bm{\theta}\leq 1\}}]=\bE_{\bm{\xi}}[\bm{v}^T\bm{\xi}]\bE_{\bm{\xi}}[1_{\{\bm{\xi}^T\bm{\theta}\leq 1\}}]=\bm{v}^T\bm{\mu}\cdot\bE_{\bm{\xi}}[1_{\{\bm{\xi}^T\bm{\theta}\leq 1\}}].
\end{equation}
Here we used that $\bm{\xi}^T\bm{v}$ and $\bm{\xi}^T\bm{\theta}$ are independent random variables and the expectation of the product of two uncorrelated random variables is the product of the expectations.  Now note that for any $\bm{\theta}$, the quantity  $\bE_{\bm{\xi}}[1_{\{\bm{\xi}^T\bm{\theta}\leq 1\}}]$ is strictly positive. Therefore, if  $\bm{v}^T\bm{\theta}=0$ and $\nabla f(\bm{\theta})=\bm{0}$ then, using \eqref{eq:hinge_min_lemma}, we obtain that $\bm{v}^T\bm{\mu}=0$. Hence, we established that $\nabla f(\bm{\theta})=\bm{0}$ implies $\bm{\theta}=\rho \bm{\mu}$ for some $\rho \in \R$.  On the other hand, using \eqref{eq:hinge_min_lemma} again, we have that $\nabla f(\rho \bm{\mu})=0$ if and only if $\bm{\mu}^T\nabla f(\rho \bm{\mu})=0$. To see the only if direction, suppose $\bm{\mu}^T\nabla f(\rho \bm{\mu}) = 0$ and $\nabla f(\rho \bm{\mu}) \neq 0$. Then we have $\nabla f(\rho \bm{\mu}) = \bm{v}$ where the vector $\bm{v}$ is nonzero such that $\bm{v}^T\bm{\mu} = 0$. By \eqref{eq:hinge_min_lemma}, we deduce $\norm{\bm{v}}^2 = \bm{v}^T \nabla f(\rho \bm{\mu}) = 0$ yielding a contradiction.

Next, consider the function
\begin{equation}\label{eq:derivative_hinge}
   g(\rho)=\bE_{\bm{\xi}}[ \bm{\mu}^T\bm{\xi} 1_{\{\rho \bm{\xi}^T\bm{\mu}\leq 1 \}}].
\end{equation}

Observe that $g(\rho) = \bm{\mu}^T \nabla f(\rho \bm{\mu})$. Dominated Convergence Theorem yields that
\[
\lim_{\rho \to +\infty}g(\rho)=\bE_{\bm{\xi}}[\bm{\mu}^T\bxi1_{\{\bm{\mu}^T\bm{\xi}\leq 0 \}}], \quad \lim_{\rho \to -\infty}g(\rho)=\bE_{\bm{\xi}}[\bm{\mu}^T\bxi1_{\{\bm{\mu}^T\bm{\xi}\geq 0 \}}].
\]
It, therefore, holds that $\lim_{\rho \to +\infty}g(\rho)<0$ and $\lim_{\rho \to -\infty}g(\rho)>0$. Since $g(0)=\bE_{\bm{\xi}}[\bm{\mu}^T\bm{\xi}]>0$, it remains to show that $g$ is a strictly decreasing function. To this end, we note that for any fixed $\rho_1<\rho_2$, it holds that
\begin{equation}\label{eq:hinge_g_decreasing}
\bm{\mu}^T\bm{\xi}\left(1_{\{\rho_1\bm{\mu}^T\bm{\xi}\leq 1 \}}-1_{\{\rho_2\bm{\mu}^T\bm{\xi}\leq 1 \}} \right) \geq 0 \quad \text{for any value of $\bm{\xi}$}.
\end{equation}
Indeed, if $\bm{\mu}^T\bm{\xi}\geq 0$, then $\rho_1\bm{\mu}^T\bm{\xi}\leq \rho_2\bm{\mu}^T\bm{\xi}$; thus ensuring $1_{\{\rho_1\bm{\mu}^T\bm{\xi}\leq 1 \}}\geq 1_{\{\rho_2\bm{\mu}^T\bm{\xi}\leq 1 \}}$. The case $\bm{\mu}^T\bm{\xi}\leq 0$ follows similarly. We, therefore, conclude that $g(\rho_1)\geq g(\rho_2)$. Finally, note that $g(\rho_1)=g(\rho_2)$, implies that \eqref{eq:hinge_g_decreasing} holds with equality, almost surely. Clearly, this yields a contradiction. It remains to show \eqref{eq:w_hinge}. By \eqref{eq:derivative_hinge}, we have that $g'(\rho^*)=\bE_{\bm{\xi}}[\bm{\mu}^T\bxi1_{\{\bm{\mu}^T\bm{\xi}\leq \frac{1}{\rho^*}\}}]$.  Using \eqref{eq:fact_affine} and \eqref{eqn:fact_truncated}, we obtain that
\begin{equation}\label{myeq2}
    \Phi\left(\frac{1-\rho^*\Vert \bm{\mu} \Vert^2}{\rho^*\sigma\Vert\bm{\mu}\Vert} \right)\cdot \exp\left(\frac{1}{2}\cdot\left(\frac{1-\rho^*\Vert \bm{\mu} \Vert^2}{\rho^*\sigma\Vert\bm{\mu}\Vert} \right)^2\right)=\frac{1}{\sqrt{2\pi}}\cdot\frac{\sigma}{\Vert\mu\Vert}.
\end{equation}
The result immediately follows.

\end{proof}

The previous lemma has defined $\bm{\theta}^*$ to be the minimizer of the loss function and showed that it is a positive multiple of $\bm{\mu}$.  We now show that this $\bm{\theta}^*$ and its positive scalar multiples are exactly the set of optimal classifiers in the sense of \eqref{eq:stop_criteria_observation}, i.e., we give an exact characterization of that set.

\begin{lemma}[Characterization of the optimal classifier]
The following is true
\begin{equation}
\Argmax_{\bm{\theta}} \bP\left(\bm{\xi}^T\bm{\theta}>0\right) = \{ \lambda\cdot \bm{\theta}^*: \lambda>0\}.\end{equation}
\end{lemma}

\begin{proof}
Observe that the following simple fact holds.
\begin{equation}\label{app.fact:PhiProb}
    \bP_{\hat{\bm{\xi}}}\left(\hat{\bm{\xi}}^T\bm{\theta}\geq t\right)=\Phi^c\left( \frac{\bm{\mu}^T\bm{\theta}-t}{\sigma \Vert \bm{\theta}\Vert}\right), \quad \text{for all $\bm{\theta}\in \R^d$, $t\in \R$ and $\hat{\bm{\xi}} \sim N(\bm{\mu},\sigma^2I_d)$}.
\end{equation}
Therefore we have that $\bP_{\bm{\xi}}(\bm{\xi}^T\bm{\theta}>0)=\Phi^c\left(\frac{\Vert \bm{\mu} \Vert}{\sigma}\cdot\cos(w_{\bm{\theta}}) \right)$ where $\bm{\xi} \sim N(\bm{\mu},\sigma^2I_d)$ and $w_{\bm{\theta}}$ denotes the angle between the two vectors $\bm{\theta}$ and $\bm{\mu}$. On the other hand a classifier $\bm{\theta}$ is optimal if and only if $\bm{\theta}=\rho \bm{\mu}$ for some $\rho > 0$, i.e. $\cos(w_{\bm{\theta}})=0$. The proof is complete after noting that $\Phi$ is an increasing function.
\end{proof}

\subsubsection{Low Variance Regime}\label{sec:low_log}
The main goal of this subsection is to give a proof of Theorem~\ref{theorem_logistic}.1. We restate it below.
\begin{manualtheorem}{2.1 }[]\label{thm.a}
Let $\bm{\theta}_0=\bm{0}$ and suppose that $\sigma \leq 0.16\Vert \bm{\mu} \Vert$. The following is then true.
\begin{equation}\begin{aligned}
\bE[T]&\leq 4+\frac{\sigma}{\Vert \bm{\mu}\Vert}\left(\frac{29,584 \cdot (1+\alpha \Vert \bm{\mu}\Vert^2)^2}{\alpha \Vert \bm{\mu}\Vert^2}+15,224 \cdot \sigma^4\alpha^4\Vert \bm{\mu}\Vert^4\right)+\frac{14,792 \cdot (1+\alpha \Vert \bm{\mu}\Vert^2)^2}{\alpha\Vert\bm{\mu}\Vert^2}.
\end{aligned} \end{equation}
  \end{manualtheorem}
Recall that in the low variance regime (\textit{i.e.}, $\sigma \leq 0.16\Vert \bm{\mu} \Vert$), we define the target set by
\begin{equation}\label{eq:C_low_proof}
    C=\{\bm{\theta}:\bm{\theta}^T\bm{\mu}\geq 1\}
\end{equation}
and the test function by
\begin{equation}\label{eq:V_low_proof}
 V=\left(M-\bm{\mu}^T\bm{\theta}\right)^2
\end{equation}
where $M=86+86\alpha \Vert \bm{\mu} \Vert^2$. As we discussed earlier, our aim is to show that
\[
 \bE[\tau_m^C] \underset{\sim}{<} m.
\]
The following proposition bounds the expected value of the stopping time $\tau_1^C$ provided we initialize with $\bm{\theta}_0=\bm{\theta}$ such that $\bm{\mu}^T\bm{\theta}<1$.
\begin{proposition}\label{prp:low_et1} Fix a vector $\bm{\theta} \in \R^d$ such that $\bm{\mu}^T \bm{\theta} < 1$. Suppose that $\sigma\leq 0.16 \Vert \bm{\mu} \Vert$. Consider the set $C$ and the function $V$ defined in \eqref{eq:C_low_proof} and \eqref{eq:V_low_proof} respectively. Then the following bound holds
\begin{equation}
    \bE[\tau_1^{C}|\bm{\theta}_0=\bm{\theta}] \leq \frac{V(\bm{\theta})}{\alpha \Vert \bm{\mu} \Vert^2}.
\end{equation}
\end{proposition}
\begin{proof}[Proof of Proposition \ref{prp:low_et1}] For simplicity we write $\mathcal{F}_{-1}:=\sigma\left(\{\bm{\theta}_0=\bm{\theta}\} \right)$.  By expanding out the term, we get for any $k \ge 1$ the following
\begin{align} \label{eq: low_noise_blah_1}
    V(\bm{\theta}_{k})= V(\bm{\theta}_{k-1})-\frac{2\alpha \bm{\mu}^T\bm{\xi}_k(M-\bm{\mu}^T\bm{\theta}_{k-1})}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}+\frac{\alpha^2(\bm{\mu}^T\bm{\xi}_k)^2}{(1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1}))^2}.
\end{align}
We fix an integer $n \ge 1$ and write the quantity $\bm{\mu}^T\bm{\xi}_k=\Vert\bm{\mu}\Vert^2+\sigma\Vert \bm{\mu}\Vert \psi_k$ with $\psi_k \sim N(0,1)$. We have
\begin{align*}
 &\bE_{\bm{\xi}_k}\left[ \frac{\bm{\mu}^T\bm{\xi}_k}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
 & \qquad =\bE_{\bm{\xi}_k}\left[ \frac{\Vert\bm{\mu}\Vert^2}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}+\sigma \Vert \bm{\mu}\Vert\bE_{\bm{\xi}_k}\left[ \frac{\psi_k}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
 &  \qquad \geq \Vert \bm{\mu} \Vert^2 \bE_{\bm{\xi}_k}\left[ \frac{1}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})}|\mathcal{F}_{k-1}\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}-\sigma \Vert \bm{\mu}\Vert\bE_{\psi_k}\left[ \vert \psi_k \vert\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
  &  \qquad \ge \Vert \bm{\mu} \Vert^2 \bE_{\bm{\xi}_k}\left[ \frac{1}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})} \left ( 1_{\{\bm{\mu}^T \bm{\theta}_{k-1} \geq  \bm{\xi}_k^T \bm{\theta}_{k-1}\}} + 1_{\{\bm{\mu}^T \bm{\theta}_{k-1} < \bm{\xi}_k^T \bm{\theta}_{k-1}\}} \right ) |\mathcal{F}_{k-1}\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
  & \qquad \qquad \qquad -\sigma \Vert \bm{\mu}\Vert \sqrt{\frac{2}{\pi}}1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
 &  \qquad \geq \frac{\Vert \bm{\mu}\Vert^2}{1+\exp(\bm{\mu}^T\bm{\theta}_{k-1})}\bE_{\bm{\xi}_k}\left[ 1_{\{\bm{\mu}^T \bm{\theta}_{k-1} \geq  \bm{\xi}_k^T \bm{\theta}_{k-1}\}} |\mathcal{F}_{k-1} \right]1_{\{ \tau_1^{C}\wedge n \geq k\}}-\sigma \Vert \bm{\mu}\Vert \sqrt{\frac{2}{\pi}}1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
 &\qquad \geq \frac{\Vert \bm{\mu}\Vert^2}{2(1+e)}1_{\{ \tau_1^{C}\wedge n \geq k\}}-\sigma \Vert \bm{\mu}\Vert \sqrt{\frac{2}{\pi}}1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
 & \qquad \geq 0.006\Vert \bm{\mu} \Vert^2 1_{\{ \tau_1^{C}\wedge n \geq k\}}.
\end{align*}
Here the first inequality follows from $-\bE[X] \le \bE[|X|]$ and $1+ \exp(\bm{\xi}_k^T\bm{\theta}_{k-1}) \ge 1$, the second from ~\eqref{fact:norm_Gaussians}, and the second to last from the observation that for any $X$ normally distributed, $\bP(\bE[X] \ge X) = 1/2$ and $\bm{\xi}_k^T\bm{\theta}_{k-1} \sim N(\bm{\mu}^T\bm{\theta}_{k-1}, \sigma^2 \norm{\bm{\theta}_{k-1}}^2)$ and $\bm{\mu}^T\bm{\theta}_{k-1} < 1$ on the set $\tau_1^{C} \wedge n \ge k$. The last inequality uses the assumption $\sigma \le 0.16 \norm{\bm{\mu}}$. By taking the conditional expectations of \eqref{eq: low_noise_blah_1} combined with the above sequence of inequalities, we deduce the following bound
\begin{align*}
    &\bE\left[V(\bm{\theta}_{k})-V(\bm{\theta}_{k-1})|\mathcal{F}_{k-1}\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
    & \quad \qquad = \bE_{\bm{\xi}_k} \left [-\frac{2\alpha \bm{\mu}^T\bm{\xi}_k(M-\bm{\mu}^T\bm{\theta}_{k-1})}{1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1})} | \mathcal{F}_{k-1} \right] 1_{\{ \tau_1^{C}\wedge n \geq k\}} + \bE_{\bm{\xi}_k} \left [\frac{\alpha^2(\bm{\mu}^T\bm{\xi}_k)^2}{(1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1}))^2} | \mathcal{F}_{k-1} \right] 1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
    & \quad \qquad \leq -0.012(M-1)\alpha \Vert \bm{\mu} \Vert^21_{\{ \tau_1^{C}\wedge n \geq k\}}+ \alpha^2\Vert \bm{\mu} \Vert^2\left(\Vert \bm{\mu}\Vert^2+\sigma^2 \right)1_{\{ \tau_1^{C}\wedge n \geq k\}}\\
    & \quad \qquad = \alpha \Vert \bm{\mu} \Vert^2\left[ \alpha\left(\Vert \bm{\mu} \Vert^2+\sigma^2\right)-0.012(M-1)\right]1_{\{ \tau_1^{C}\wedge n \geq k\}}.
\end{align*}
Here the first inequality follows from $\bm{\mu}^T\bm{\theta}_{k-1} 1_{\{\tau_1^{C} \wedge n \ge k\}} \le 1_{\{\tau_1^{C}\wedge n \geq k\}}$ and upper bounding $\frac{(\bm{\mu}^T\bm{\xi}_k)^2}{(1+\exp(\bm{\xi}_k^T\bm{\theta}_{k-1}))^2}$ by $(\bm{\mu}^T\bm{\xi}_k)^2$ and then applying \eqref{fact:norm_Gaussians}.
A quick computation after plugging in the value of $M$ and the bound $\sigma\leq 0.16\Vert \bm{\mu} \Vert$ yields
\begin{equation} \label{eq: low_noise_blah_10}
  \bE\left[V(\bm{\theta}_{k})-V(\bm{\theta}_{k-1})|\mathcal{F}_{k-1}\right]1_{\{ \tau_1^{C}\wedge n \geq k\}} \leq -\alpha \Vert \bm{\mu} \Vert^21_{\{ \tau_1^{C}\wedge n \geq k\}}.
\end{equation}
Using Appendix Lemma \ref{lem:drift_lemma_appendix}, we obtain that $\bE[\tau_1^{C}\wedge n|\mathcal{F}_{-1}] \leq \frac{V(\bm{\theta})}{\alpha \Vert \bm{\mu} \Vert^2}$. By monotone convergence theorem the result follows.
\end{proof}
We next upper bound $\bE[\tau_m^C]$ for $m\geq 1$ in the following proposition.
\begin{proposition}\label{prp:low_etm}
Let $\bm{\theta}_0=\bm{0}$ and suppose that $\sigma\leq 0.16 \Vert \bm{\mu} \Vert$. Consider the set $C$ defined in \eqref{eq:C_low_proof}. Then the following bound holds for all $m \ge 1$
\begin{equation}
   \bE[\tau_m^C]\leq (m-1)\left(2+\frac{\sigma}{\Vert \bm{\mu}\Vert}\left(\frac{2M^2}{\alpha \Vert \bm{\mu}\Vert^2}+7,612 \cdot \sigma^4\alpha^4\Vert \bm{\mu}\Vert^4\right)\right)+\frac{M^2}{\alpha \Vert \bm{\mu}\Vert^2},
\end{equation}
where $M=86+86\alpha \Vert \bm{\mu} \Vert^2$.
\end{proposition}
\begin{proof}[Proof of Proposition \ref{prp:low_etm}]
First, it is clear when $m = 1$, the result holds by Proposition~\ref{prp:low_et1} with $\bm{\theta}_0 = \bm{0}$. We now assume that $\tau_{m-1}^{C}<\infty$ a.s. for some $m\geq 2$ and fix an integer $n \ge 1$. We decompose the space to yield the following bounds

 \begin{equation} \begin{aligned} \label{eq: low_noise_bound}
    \bE\left[(\tau_m^{C}-\tau_{m-1}^{C})\wedge n|\mathcal{F}_{\tau_{m-1}^{C}+1}\right]&=\bE\left[((\tau_m^{C}-\tau_{m-1}^{C})\wedge n)|\mathcal{F}_{\tau_{m-1}^{C}+1}\right]1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}\geq 1\}}\\
    &+\bE\left[((\tau_m^{C}-\tau_{m-1}^{C})\wedge n)|\mathcal{F}_{\tau_{m-1}^{C}+1}\right]1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<1\}}\\
    &\leq 1+ \bE\left[((\tau_m^{C}-\tau_{m-1}^{C})\wedge n)|\mathcal{F}_{\tau_{m-1}^{C}+1}\right]1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<1\}}\\
    &=1+\sum_{i=1}^{\infty}\bE\left[(\tau_m^{C}-\tau_{m-1}^{C})\wedge n|\mathcal{F}_{\tau_{m-1}^{C}+1}\right]1_{\{i-1\leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<i\}}\\
&\leq 2+\sum_{i=1}^{\infty}\bE\left[\tau_1^{C}\wedge n|\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}^{C}+1}\right]1_{\{i-1\leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<i\}}.
\end{aligned} \end{equation}
Here the first inequality follows because $((\tau_m^{C}-\tau_{m-1}^{C})\wedge n) 1_{\{\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1} \ge 1\}} \le 1$ by definition of C and the last inequality by the strong Markov property. For each $i \ge 1$, we observe the bound
\begin{align}
  1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1} &=1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}}-\frac{\alpha \bm{\mu}^T\bm{\xi}_{{\tau_{m-1}^{C}}+1}}{1+\exp(\bm{\xi}_{{\tau_{m-1}^{C}}+1}^T\bm{\theta}_{\tau_{m-1}^{C}})}\nonumber\\
    &\leq -\frac{\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^{C}+1}}{1+\exp(\bm{\xi}_{\tau_{m-1}^{C}+1}^T\bm{\theta}_{\tau_{m-1}^{C}})} \label{eq:i-1_calculus}\\
    & \leq -\alpha \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^{C}+1},\nonumber
\end{align}
where the second inequality follows because $1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}} \le 0$ and the last inequality because $-\alpha \bm{\mu}^T \bm{\xi}_{\tau_{m-1}^{C}+1}$ is positive.
By Proposition~\ref{prp:low_et1} and the above, we deduce
\begin{equation} \begin{aligned}  \label{eq: low_noise_blah_3}
\bE\left[\tau_1^{C}\wedge n|\bm{\theta}_0=\bm{\theta}_{\tau_{m-1}^{C}+1}\right]1_{\{i-1\leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}<i\}}&\leq  \frac{(M-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1})^2}{\alpha \Vert \bm{\mu} \Vert^2}1_{\{i-1\leq 1-\bm{\mu}^T\bm{\theta}_{\tau_{m-1}^{C}+1}\leq i\}}\\&\leq  \frac{(M+i-1)^2}{\alpha \Vert \bm{\mu}\Vert^2}1_{\{ \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^{C}+1}\leq \frac{1-i}{\alpha}\}}.
\end{aligned} \end{equation}
Finally we observe that
\begin{equation} \begin{aligned} \label{eq: low_noise_blah_4}
    \bE\left[1_{\{ \bm{\mu}^T\bm{\xi}_{\tau_{m-1}^{C}+1}\leq \frac{1-i}{\alpha}\}} \right]&=\bE\left[\sum_{k=1}^{\infty}1_{\{ \bm{\mu}^T\bm{\xi}_{k+1}\leq \frac{1-i}{\alpha}\}}1_{\{\tau_{m-1}^{C}=k\}}\right]\\
&=\sum_{k=1}^{\infty}\bE\left[1_{\{ \bm{\mu}^T\bm{\xi}_{k+1}\leq \frac{1-i}{\alpha}\}}  \right]\bE\left[1_{\{\tau_{m-1}^{C}=k\}} \right]\\
    &= \Phi\left(\frac{\frac{1-i}{\alpha}-\Vert \bm{\mu}\Vert^2}{\sigma\Vert \bm{\mu}\Vert } \right)\sum_{k=1}^{\infty}\bE\left[1_{\{\tau_{m-1}^{C}=k\}} \right]\\
    &=\Phi\left(\frac{\frac{1-i}{\alpha}-\Vert \bm{\mu}\Vert^2}{\sigma\Vert \bm{\mu}\Vert } \right).
\end{aligned} \end{equation}
The first equality is by independence and the second equality because $\bm{\mu}^T \bm{\xi}_{k+1} \sim N(\norm{\bm{\mu}}^2, \sigma^2 \norm{\bm{\mu}}^2)$. Indeed, the same argument shows $\bm{\xi}_{\tau_{m-1}^{C}+1} \sim N(\bm{\mu},\sigma^2I_d)$, see \cite[Theorem 4.3.1]{Durrett_probability_book}. By combining  \eqref{eq: low_noise_bound}, \eqref{eq: low_noise_blah_3}, and \eqref{eq: low_noise_blah_4}, we obtain the following
\begin{equation} \begin{aligned} \label{eq: low_noise_blah_5}
    \bE\left[(\tau_m^{C}-\tau_{m-1}^{C})\wedge n\right]&\leq 2+\sum_{i=1}^{\infty} \frac{(M+i-1)^2}{\alpha \Vert \bm{\mu}\Vert^2}\Phi\left(\frac{\frac{1-i}{\alpha}-\Vert \bm{\mu}\Vert^2}{\sigma\Vert \bm{\mu}\Vert } \right)\\&=2+\sum_{i=1}^{\infty} \frac{(M+i-1)^2}{\alpha \Vert \bm{\mu}\Vert^2}\Phi^{c}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert } \right)\\&\le2+\frac{\sigma}{\Vert \bm{\mu}\Vert\sqrt{2\pi}}\sum_{i=1}^{\infty} \frac{(M+i-1)^2}{\alpha\Vert \bm{\mu}\Vert^2+i-1}\exp\left(-\frac{1}{2}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert}\right)^2 \right),
\end{aligned} \end{equation}
where we used the inequality $\Phi^{c}(t)<\frac{1}{t\sqrt{2\pi}}\exp(-\frac{t^2}{2})$. We bound the sum by
\begin{equation}
\begin{aligned} \label{eq: low_noise_blah_6}
    \sum_{i=3}^{\infty} &\frac{(M+i-1)^2}{\alpha\Vert \bm{\mu}\Vert^2+i-1}\exp\left(-\frac{1}{2}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert}\right)^2 \right)\\
    &\qquad = \sum_{i=3}^{\infty} \frac{(M+i-1)^2}{\alpha\Vert \bm{\mu}\Vert^2+i-1} \cdot \frac{4\sigma^4 \norm{\bm{\mu}}^2 \alpha^4}{(\alpha \norm{\bm{\mu}}^2 + i-1)^4} \cdot \frac{(\alpha \norm{\bm{\mu}}^2 + i-1)^4}{4\sigma^4 \norm{\bm{\mu}}^2 \alpha^4} \exp\left(-\frac{1}{2}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert}\right)^2 \right)\\
    &\qquad \leq 4\alpha^4\sigma^4\Vert \bm{\mu} \Vert^4  \sum_{i=3}^{\infty}\frac{(M+i-1)^2}{(\alpha \Vert \bm{\mu}\Vert^2+i-1)^4}\\
    &\qquad \leq 29,584 \cdot \alpha^4\sigma^4\Vert \bm{\mu}\Vert^4 \sum_{i=3}^{\infty}\frac{1}{(i-1)^2} \\
    &\qquad \leq 29,584 \cdot \left ( \frac{\pi^2}{6}-1 \right ) \cdot \alpha^4\sigma^4\Vert \bm{\mu}\Vert^4\\
    & \qquad \leq 19,080 \cdot \alpha^4\sigma^4 \Vert \bm{\mu}\Vert^4.
\end{aligned} \end{equation}
Here the first inequality uses $x^2\leq e^x$ for all $x>0$ and $\alpha \norm{\bm{\mu}}^2 + i-1 \ge 1$ for all $i \ge 2$, the second uses that $\frac{M+i-1}{\alpha \Vert \bm{\mu}\Vert^2+i-1}\leq 86$ for all $i \ge 3$ and  $\frac{1}{\alpha \Vert \bm{\mu}\Vert^2+i-1}\leq \frac{1}{i-1}$ for $i\geq 2$. We also upper bound

\begin{align} \label{eq: low_noise_blah_7}
  \sum_{i=1}^{2} \frac{(M+i-1)^2}{\alpha\Vert \bm{\mu}\Vert^2+i-1}\exp\left(-\frac{1}{2}\left(\frac{\Vert \bm{\mu}\Vert^2+\frac{i-1}{\alpha}}{\sigma\Vert \bm{\mu}\Vert}\right)^2 \right) &\leq \frac{M^2}{\alpha \Vert \bm{\mu} \Vert^2}+\frac{(M+1)^2}{\alpha \Vert \bm{\mu} \Vert^2+1} \le \frac{5M^2}{\alpha \norm{\bm{\mu}}^2},
\end{align}
where the first inequality we use $e^x \le 1$ for all $x<0$ and the second inequality because $\alpha \norm{\bm{\mu}}^2 + 1 \ge \alpha \norm{\bm{\mu}}^2$ and $M +1 \le 2M$. By combining \eqref{eq: low_noise_blah_5}, \eqref{eq: low_noise_blah_6}, and \eqref{eq: low_noise_blah_7}, we obtain
\begin{align*}
   \bE\left[(\tau_m^{C}-\tau_{m-1}^{C})\wedge n\right]&\leq 2+\frac{\sigma}{\Vert \bm{\mu}\Vert}\left(\frac{2M^2}{\alpha \Vert \bm{\mu}\Vert^2}+7,612 \cdot \sigma^4\alpha^4\Vert \bm{\mu}\Vert^4\right).
   \end{align*}
Taking the limit as $n \to +\infty$, we observe that
\[\bE[\tau_m^C]\leq 2+\frac{\sigma}{\Vert \bm{\mu}\Vert}\left(\frac{2M^2}{\alpha \Vert \bm{\mu}\Vert^2}+7,612 \cdot \sigma^4\alpha^4\Vert \bm{\mu}\Vert^4\right)+\bE[\tau_{m-1}^{C}].
\]
We then iterate the above inequality yielding
\[ \bE[\tau_m^C] \le (m-1) \left ( 2+\frac{\sigma}{\Vert \bm{\mu}\Vert}\left(\frac{2M^2}{\alpha \Vert \bm{\mu}\Vert^2}+7,612 \cdot \sigma^4\alpha^4\Vert \bm{\mu}\Vert^4\right)  \right ) + \bE[\tau_1^{C}]. \]
The result follows after applying Proposition~\ref{prp:low_et1} with $\bm{\theta}_0 = \bm{0}$.
\end{proof}
We are now ready to prove Theorem~\ref{thm.a}.
\begin{proof}[Proof of Theorem
\ref{thm.a}]
In order to simplify the subsequent argument, we define the quantity,
\[ M' := 2+\frac{\sigma}{\Vert \bm{\mu}\Vert}\left(\frac{2M^2}{\alpha \Vert \bm{\mu}\Vert^2}+7,612\sigma^4\alpha^4\Vert \bm{\mu}\Vert^4\right) \]
where $M = 86 + 86 \alpha \norm{\bm{\mu}}^2$, we see in Proposition~\ref{prp:low_etm}.  By Lemma \ref{lem:st_ETC},  $T_C<\infty$ a.s. and, therefore we have $\sum_{m=1}^{\infty}1_{\{T_C=\tau_m^{C}\}} = 1$ a.s. We next apply Proposition~\ref{prp:low_etm} with Lemma \ref{lem:st_ETC}. Note that $\delta:=\tfrac{1}{2}$ with the target set $C=\{\bm{\theta}:\bm{\mu}^T\bm{\theta}\geq 1 \}$ satisfy \eqref{eq:ST_delta}
\begin{align*}
\bE[T]&\leq \bE[T_C]=\sum_{m=1}^{\infty} \bE[\tau_C1_{\{T_C=\tau_m^{C}\}}]\leq \sum_{m=1}^{\infty}\frac{\bE[\tau_m^C]}{2^{m-1}} \le \sum_{m=1}^\infty \frac{(m-1)M' + \frac{M^2}{\alpha \norm{\bm{\mu}}^2} }{2^{m-1}} = 2M' + \frac{2M^2}{\alpha \norm{\bm{\mu}}^2}.
\end{align*}
The result follows after plugging in the value of $M = 86 + 86 \alpha \norm{\bm{\mu}}^2$.
\end{proof}

\subsubsection{High Variance Regime}\label{sec:high_logistic}
The main goal of this subsection is to give a proof of Theorem~\ref{theorem_logistic}.2. We restate it below.
\begin{manualtheorem}{2.2}[]\label{thm.b}
Let $\bm{\theta}_0=\bm{0}$ and suppose that $ 0.16\Vert \bm{\mu} \Vert \leq \sigma$. Then it holds that $\bE[T]<+\infty$ provided the step-size $\alpha$ satisfies
\begin{equation}\label{eq:high_alpha}
\alpha < \tfrac{\min\{1,B\}}{\Vert \bm{\mu} \Vert^2+d\sigma^2}
\end{equation}
where the constant $B$ is defined by
\begin{equation}
B:=\tfrac{\Vert \bm{\mu}\Vert^4}{4\sigma^4\sqrt{\pi}}\exp\left(-\tfrac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}\right) \Phi^{c}\left( \tfrac{7\sqrt{2}\Vert \bm{\mu} \Vert}{\sigma}\right).
\end{equation}
  \end{manualtheorem}
The proof is deferred to the end of this subsection. We first need some preliminaries. Throughout this subsection, the following notation is used
\[
 \bm{\theta} = \rho\bm{\mu}+ \bm{\tilde{\theta}} \text{ where } \bm{\mu}^T \bm{\tilde{\theta}} = 0.
\]
We also express $\bm{\theta}_k=\rho_k\bm{\mu}+\tilde{\bm{\theta}}_k$, etc.  Recall that in the high variance regime (\textit{i.e.}, $0.16\Vert \bm{\mu} \Vert\leq \sigma$), we define the target set by
\begin{equation}\label{eq:C_high_proof_1}
     C=\left\{\bm{\theta}: \vert \rho\sigma^2-2\vert < 1 \text{ and } \sigma \Vert \tilde{\bm{\theta}} \Vert \leq 3404.46 \right\}.`
\end{equation}
We first establish \eqref{eq:ST_delta}.

\begin{lemma}\label{lem: high_noise_delta}
Consider the set defined in \eqref{eq:C_high_proof_1}. Fix $\bm{\theta}\in {C}$ and let $\hat{\bm{\xi}} \sim N(\bm{\mu},\sigma^2 I_d)$. The following bound holds.
\begin{equation}\label{eq:deltaP}
    \bP_{\hat{\bm{\xi}}}\left(\bm{\theta}^T\hat{\bm{\xi}}\geq 1\right)\geq \Phi\left(\frac{-\sigma}{\Vert \bm{\mu} \Vert}\right).
\end{equation}
\end{lemma}

\begin{proof}
It is easy to verify that $\bP_{\hat{\bm{\xi}}}\left(\bm{\theta}^T\hat{\bm{\xi}}\geq 1\right)=\Phi\left(\frac{\bm{\theta}^T\bm{\mu}-1}{\sigma \Vert \bm{\theta}\Vert} \right)$. We are interested in lower bounding the term $\frac{\bm{\theta}^T\bm{\mu}-1}{\sigma \Vert \bm{\theta}\Vert}$ independent of $\bm{\theta}$.

As $\bm{\theta} \in {C}$, we have $\vert \rho\sigma^2-2\vert < 1$ which yields $\rho \geq \frac{1}{\sigma^2}$. Combining this with $\Vert \bm{\theta} \Vert^2 = \| \rho \bm{\mu} + \tilde{\bm{\theta}} \|^2 \geq \rho^2\Vert \bm{\mu} \Vert^2$, we obtain that $\sigma \Vert \bm{\theta} \Vert\geq \frac{\Vert \bm{\mu} \Vert}{\sigma}$. This gives
\begin{equation}
     \frac{\bm{\theta}^T\bm{\mu}-1}{\sigma \Vert \bm{\theta}\Vert} = \frac{(\rho \bm{\mu} + \tilde{\bm{\theta}})^T\bm{\mu}-1}{\sigma \norm{\bm{\theta}}} \geq \frac{-1}{\sigma \Vert \bm{\theta}\Vert}\geq  \frac{-\sigma}{\Vert \bm{\mu} \Vert}.
\end{equation}
Here the first inequality follows from $\rho>0$ and the second inequality from $\sigma \Vert \bm{\theta} \Vert\geq \frac{\Vert \bm{\mu} \Vert}{\sigma}$. The proof is complete after noting that $\Phi$ is an increasing function.
\end{proof}
We need the following technical lemma below. The proof is deferred to Appendix Sec. ???.

\begin{lemma}\label{lem:high_tech_not_appendix}
Consider the following optimization problem
\[
\min f(\bm{\theta}):=\bE_{(\bm{\xi},y)\sim \mathcal{P}_*}\left[\ell(\bm{\xi}^T\bm{\theta},y)\right].
\]
where $\ell$ is a convex function. The sequences $\{\bm{\theta}_k, \bm{\xi}_k \}_{k = 0}^\infty$ generated by Algorithm~\ref{alg:SGD_termination} satisfy for any $\bm{\theta} \in \R^d$, the following
    \begin{equation} \label{eq:tech_lemma_inside}
       f(\bm{\theta}_{k-1})-f(\bm{\theta})\leq \frac{1}{2\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta} \Vert^2-\bE\left[\Vert \bm{\theta}_{k}-\bm{\theta} \Vert^2\, |\mathcal{F}_{k-1}\right] \right)+\bE[\Vert\nabla_{\bm{\theta}}\ell\left( \bm{\xi}_k^T\bm{\theta}\right)\Vert^2 \, | \, \mathcal{F}_{k-1}],
    \end{equation}
    for all $k\geq 1$ where $f$ is defined in \eqref{optimization_problem} and the filtration $\{\mathcal{F}_k\}_{k=0}^{+\infty}$ in \eqref{eq:sigma}.
\end{lemma}
\begin{remark}
Note that in \eqref{eq:tech_lemma_inside}, we are not assuming that $\ell$ is differentiable. We have two choices for the $\ell$ function: logistic loss and hinge loss. Nevertheless, hinge loss is not differentiable, since for any $\bm{\theta}$ w.p. 1 $\bm{\xi}_k^T\bm{\theta}\neq 1$, we conclude that $\ell(\bm{\xi}_k^T\bm{\theta})$ as a function of $\bm{\xi}_k$ is almost everywhere differentiable.
\end{remark}

Let $\bm{\theta}$ be $\bm{\theta}^*$  in Lemma~\ref{lem:high_tech_not_appendix} and write
\begin{equation}\label{eq:high_ftheta_ktheta}
    f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)=f(\bm{\theta}_{k-1})-f(\rho\bm{\mu})+f(\rho\bm{\mu})-f(\bm{\theta}^*).
\end{equation}
In Lemmas~\ref{lem: logistic_case_2} and \ref{lem: logistic_case_1} below, we will show that both terms $f(\bm{\theta}_{k-1})-f(\rho \bm{\mu})$ and $f(\rho\bm{\mu})-f(\bm{\theta}^*)$ are non-negative. Moreover, we will establish that whenever $\bm{\theta}_{k-1} \not\in C$, either the first or second term can be lower bounded by some positive constant.  We first lower bound $f(\bm{\theta}_{k-1})-f(\rho \bm{\mu})$.  Let us recall that $\bm{\theta}=\rho \bm{\mu}+\tilde{\bm{\theta}}$ where $\hat{\bm{\theta}}^T\bm{\mu}=0$.
\begin{lemma} \label{lem: logistic_case_2} Fix a vector $\bm{\theta} \in \R^d$ and let $\bm{\xi} \sim N(\bm{\mu},\sigma^2I_d)$. Provided that $\sigma \Vert \tilde{\bm{\theta}}\Vert\geq 4\log(\sqrt{2}+1)$, the following holds
\begin{equation}\label{eq:high_lem_sigmatheta}
    \bE_{\bm{\xi}} \left[\log\left( \frac{1+\exp(-\bm{\theta}^T\bm{\xi})}{1+\exp(-\rho \bm{\mu}^T\bm{\xi})}\right)\right] \geq \max\left\{\left ( \tfrac{1}{2} \log(2) + \tfrac{\sigma \Vert \tilde{\bm{\theta}}\Vert}{8} \right )\Phi^c(1)-\left | \frac{\rho}{4}\right | \left(\Vert \bm{\mu}\Vert^2+\sigma \Vert \bm{\mu} \Vert \sqrt{\frac{2}{\pi}} \right)
    ,0\right\}.
\end{equation}

\end{lemma}
\begin{proof} The two normal random variables, $\bm{\tilde{\theta}}^T\bm{\xi} \sim N(0,\sigma^2\Vert \bm{\tilde{\theta}} \Vert^2)$ and $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$, are independent by \eqref{eqn:fact_independence}. Since we have $\bE_{\bm{\xi}}[\log(\exp(-\bm{\tilde{\theta}}^T\bm{\xi}))]=\bE_{\bm{\xi}}[\log(\exp(\bm{\tilde{\theta}}^T\bm{\xi}))]=0$, it holds
\begin{align*}
    \bE_{\bm{\xi}} \left[ \log\left (1+\exp(-\bm{\theta}^T\bm{\xi})\right)\right] &= \bE_{\bm{\xi}} \left[\log \left(1 + \exp(-\tilde{\bm{\theta}}^T\bm{\xi}) \exp(-\rho \bm{\mu}^T\bm{\xi}) \right ) \right ]\\
    &=\bE_{\bm{\xi}} \left[ \log\left (\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right]\\
    &=\bE_{\bm{\xi}} \left[ \log\left (\exp(-\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right],
\end{align*}
where the last equality is true because $\bm{\tilde{\theta}}^T\bm{\xi} \sim -\bm{\tilde{\theta}}^T\bm{\xi}$. Therefore we obtain
\begin{align*}
  \bE_{\bm{\xi}} &\left[ \log\left (1+\exp(-\bm{\theta}^T\bm{\xi})\right)\right]\\
  & \qquad \qquad= \frac{1}{2} \bE_{\bm{\xi}} \left[ \log\left (\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right] + \frac{1}{2}\bE_{\bm{\xi}} \left[ \log\left (\exp(-\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)\right] \\
  & \qquad \qquad= \frac{1}{2} \bE_{\bm{\xi}} \left[ \log\left ( (\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi}))(\exp(-\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\rho \bm{\mu}^T\bm{\xi})) \right)\right]\\
  &\qquad \qquad=\frac{1}{2}\bE_{\bm{\xi}} \left[\log\left(1+\exp(-\bm{\tilde{\theta}}^T\bm{\xi}-\rho \bm{\mu}^T\bm{\xi})+\exp(\bm{\tilde{\theta}}^T\bm{\xi}-\rho \bm{\mu}^T\bm{\xi})+\exp(-2\rho \bm{\mu}^T\bm{\xi})\right) \right].
  \end{align*}
By the equality $\exp(\bm{\tilde{\theta}}^T\bm{\xi})+\exp(-\bm{\tilde{\theta}}^T\bm{\xi})= 2+4\sinh^2(\tfrac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})$, we have
\begin{align*}
   \bE_{\bm{\xi}} &\left[ \log\left (1+\exp(-\bm{\theta}^T\bm{\xi})\right)\right]\\
    & \qquad \qquad \qquad =\frac{1}{2}\bE_{\bm{\xi}} \left[\log\left(1+2\exp(-\rho \bm{\mu}^T\bm{\xi})+\exp(-2\rho \bm{\mu}^T\bm{\xi})+4\sinh^2(\tfrac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\exp(-\rho \bm{\mu}^T\bm{\xi})\right) \right].
\end{align*}
Therefore, we have
\begin{equation} \begin{aligned} \label{eqn: high_noise_blah_20}
     2\bE_{\bm{\xi}}\left[\log\left( \frac{1+\exp(-\bm{\theta}^T\bm{\xi})}{1+\exp(-\rho \bm{\mu}^T\bm{\xi})}\right)\right]&=2\bE_{\bm{\xi}}\left[\log(1+\exp(-\bm{\theta}^T\bm{\xi}))\right]-\bE_{\bm{\xi}}\left[\log\left(1+\exp(-\rho \bm{\mu}^T\bm{\xi})\right)^2\right]\\
     \qquad \qquad &= \bE_{\bm{\xi}}\left[\log \left ( 1 +  \frac{4\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\exp(-\rho \bm{\mu}^T\bm{\xi})}{(1+ \exp(-\rho \bm{\mu}^T\bm{\xi}))^2} \right ) \right] \ge 0.
     \end{aligned} \end{equation}
     Thus it remains to establish the other lower bound in \eqref{eq:high_lem_sigmatheta}. First, we note the following $1+ \exp(-\rho \bm{\mu}^T\bm{\xi}) = 2 \exp( -\tfrac{\rho \bm{\mu}^T\bm{\xi}}{2}) \cosh(\tfrac{\rho \bm{\mu}^T\bm{\xi}}{2})$. Let the constant $r > 0$ and consider the set $\{\bm{\xi}: |\bm{\theta}^T\bm{\xi}| > r\}$. Applying the inequality $x^2+y^2\geq 2\vert xy \vert$ and \eqref{eqn: high_noise_blah_20}, we obtain that
     \begin{align}
   2\bE_{\bm{\xi}}\left[\log\left( \frac{1+\exp(-\bm{\theta}^T\bm{\xi})}{1+\exp(-\rho \bm{\mu}^T\bm{\xi})}\right)\right]  & = \bE_{\bm{\xi}}\left[\log\left(1 + \frac{4\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\exp(-\rho \bm{\mu}^T\bm{\xi})}{(1+ \exp(-\rho \bm{\mu}^T\bm{\xi}))^2}\right) \right] \nonumber \\
     &= \bE_{\bm{\xi}}\left[\log\left(1+\frac{\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})}{\cosh^2(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})} \right) \right] \nonumber \\
     &\geq \bE_{\bm{\xi}}\left[\log\left(1+\frac{\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})}{\cosh^2(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})} \right) 1_{\{\bm{\xi}:\vert \tilde{\bm{\theta}}^T\bm{\xi}\vert \geq r\}}\right] \label{eq:high_noise_blah_21}
     \\
     &\geq \bE_{\bm{\xi}}\left[\left(\log2+\log\left( \frac{\vert\sinh(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})\vert}{\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})}\right) \right)1_{\{\bm{\xi}:\vert \tilde{\bm{\theta}}^T\bm{\xi}\vert \geq r\}}\right] \nonumber.
    \end{align}
Here  \eqref{eq:high_noise_blah_21} follows from $\log\left(1+\frac{\sinh^2(\frac{\tilde{\bm{\theta}}^T\bm{\xi}}{2})}{\cosh^2(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})} \right)$ is always positive. From \eqref{eq:fact_affine}, we have $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$ and $\tilde{\bm{\theta}}^T\bm{\xi} \sim N(0,\sigma^2\Vert \tilde{\bm{\theta}} \Vert^2)$. Thus, we write $\tilde{\bm{\theta}}^T\bm{\xi} = \sigma \| \tilde{\bm{\theta}} \| \psi$ where $\psi \sim N(0,1)$. Moreover, we have that $-\log\left(\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})\right)1_{\{\vert \tilde{\bm{\theta}}^T\bm{\xi}\vert \geq r\}} \geq -\log\left(\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})\right) $ since $\cosh(\frac{\rho}{2}\bm{\mu}^T\bm{\xi})\geq 1$ always holds. Using the inequality $\log\cosh(x)\leq \vert x\vert$ for all $x$, the following bound holds
     \begin{equation} \begin{aligned} \label{eq: high_noise_22}
       &\bE_{\bm{\xi}}\left[\log\left( \frac{1+\exp(-\bm{\theta}^T\bm{\xi})}{1+\exp(-\rho \bm{\mu}^T\bm{\xi})}\right)\right]\\
       & \quad \ge \tfrac{1}{2} \log(2) \cdot \bE_{\psi}\left[1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] + \tfrac{1}{2} \bE_{\psi}\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] - \tfrac{1}{2}\bE_{\bm{\xi}} \left [ \log( \cosh(\tfrac{\rho}{2} \bm{\mu}^T\bm{\xi})) \right ]\\
       & \quad \ge \tfrac{1}{2} \log(2) \cdot \bE_{\psi}\left[1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] + \tfrac{1}{2} \bE_{\psi}\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] - \tfrac{1}{2} \bE_{\bm{\xi}} \left [ | \tfrac{\rho}{2} \bm{\mu}^T\bm{\xi} | \right ]\\
       &\quad\geq \tfrac{1}{2} \log(2) \cdot \bE_{\psi}\left[1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right] + \tfrac{1}{2} \bE_{\psi}\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{\vert \psi\vert \geq \frac{r}{\sigma \Vert\tilde{\bm{\theta}}\Vert}\}}\right]-\left | \frac{\rho}{4}\right | \left(\Vert \bm{\mu}\Vert^2+\sigma \Vert \bm{\mu} \Vert \sqrt{\frac{2}{\pi}} \right),
    \end{aligned} \end{equation}
where the last inequality uses \eqref{fact:norm_Gaussians}. Using the inequality $\vert \sinh(x) \vert \geq \exp(\frac{\vert x\vert}{2})$ for $\vert x \vert \geq 2\log(\sqrt{2}+1)$, letting $r=4\log(\sqrt{2}+1)$, we obtain
\begin{align}
   \tfrac{1}{2} \log(2) \cdot \bE_{\psi} \big [ 1_{\{|\psi| \ge \frac{4 \log( \sqrt{2}+ 1)\}}{\sigma \norm{\tilde{\bm{\theta}}} }} \big ] &+ \tfrac{1}{2}\bE_\psi\left[\log\left\vert\sinh(\tfrac{\sigma\Vert \tilde{\bm{\theta}}\Vert\psi}{2})\right\vert1_{\{|\psi| \geq \frac{4\log(\sqrt{2}+1)}{\sigma \Vert \tilde{\bm{\theta}}\Vert}\}}\right]\nonumber\\
   &\geq \tfrac{1}{2} \log(2) \cdot \bE_{\psi} \big [ 1_{\{|\psi| \ge \frac{4 \log( \sqrt{2}+ 1)}{\sigma \norm{\tilde{\bm{\theta}}} } \}} \big ] +  \tfrac{1}{2}\bE_\psi\left[\left\vert\tfrac{\sigma \Vert \tilde{\bm{\theta}}\Vert \psi}{4} \right\vert1_{\{|\psi| \geq \frac{4\log(\sqrt{2}+1)}{\sigma \Vert \tilde{\bm{\theta}}\Vert}\}}\right]\nonumber\\
    &\geq \tfrac{1}{2} \log(2) \cdot \bE_{\psi}[1_{\{\vert\psi\vert \ge 1\}}] + \tfrac{1}{2} \bE_\psi\left[\left\vert\tfrac{\sigma \Vert \tilde{\bm{\theta}}\Vert \psi}{4} \right\vert1_{\vert\psi\vert \geq 1\}}\right] \label{eq:high_noise_blah_111} \\\nonumber
    &\geq \left ( \tfrac{1}{2} \log(2) + \frac{\sigma \Vert \tilde{\bm{\theta}}\Vert}{8} \right )\Phi^c(1).

\end{align}
Here \eqref{eq:high_noise_blah_111} follows from $1\geq \frac{4\log(\sqrt{2}+1)}{\sigma \Vert \tilde{\bm{\theta}}\Vert}
$. By combining the above inequality with \eqref{eq: high_noise_22}, the desired result holds.
\end{proof}
We now turn to lower bound the second summand in the right-hand side of \eqref{eq:high_ftheta_ktheta}, namely $f(\rho \bm{\mu})-f(\bm{\theta^*})$.
\begin{lemma} \label{lem: logistic_case_1}
Provided that $\vert \rho\sigma^2-2 \vert> 1$, the following estimate holds
\begin{equation}\label{eq:high_lemhrho}
    f(\rho \bm{\mu})-f(\bm{\theta^*}) \geq \frac{\Vert \bm{\mu}\Vert^4}{4\sigma^4\sqrt{\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}\right) \Phi^{c}\left( \frac{7\sqrt{2}\Vert \bm{\mu} \Vert}{\sigma}\right).
\end{equation}

\end{lemma}
\begin{proof} Define the function
\[
    g(\rho):=\bE_{\bm{\xi}}\left[\log\left(1+\exp(-\rho \bm{\mu}^T\bm{\xi})\right) \right], \quad \bm{\xi} \sim N(\bm{\mu},\sigma^2I_d).
\]
By Lemma~\ref{lem:minimizer_logistic}, we know that $g$ is a convex function with a unique minimizer at $\rho^*:=\frac{2}{\sigma^2}$. Observe that $f(\rho \bm{\mu})-f(\bm{\theta}^*) = g(\rho)-g(\rho^*)$; hence in order to prove \eqref{eq:high_lemhrho}, we instead aim to bound this difference in the function $g$. From \eqref{eq:fact_affine}, we have $\bm{\mu}^T\bm{\xi} \sim N(\Vert \bm{\mu} \Vert^2,\sigma^2\Vert \bm{\mu} \Vert^2)$. It thus holds
\[
 4g''(\rho)=\bE\left(\frac{(\bm{\mu}^T\bm{\xi})^2}{\cosh(\frac{\rho}{2} \bm{\mu}^T\bm{\xi})^2} \right)=\frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi }}\int_{-\infty}^{\infty}\frac{z^2}{\cosh^2(\frac{\rho z}{2})}\exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu} \Vert^2}\right)dz.
\]
Upper bounding $\cosh^2(\frac{\rho z}{2})$ by $\exp(\vert \rho z\vert)$, we next obtain
\begin{align*}
    4g''(\rho)&\geq \frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi}} \int_{-\infty}^{\infty} z^2\exp\left(-\vert \rho z \vert \right)\exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2}{2\sigma^2\Vert \bm{\mu}\Vert^2}\right)dz & \\ &=\frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi }} \int_{-\infty}^{\infty} z^2\exp\left(-\frac{(z-\Vert \bm{\mu} \Vert^2)^2+2\sigma^2\Vert \bm{\mu}\Vert^2\vert\rho z\vert}{2\sigma^2\Vert \bm{\mu}\Vert^2}\right)dz  ,&\\
    &=\frac{1}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi }}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right) \int_{-\infty}^{\infty} z^2\exp\left(-\frac{z^2-2\Vert \bm{\mu} \Vert^2 z+2\sigma^2\Vert \bm{\mu}\Vert^2\vert\rho z\vert}{2\sigma^2\Vert \bm{\mu}\Vert^2}\right)dz
\end{align*}
One can easily verify that whenever $\vert z \vert \geq 2\Vert \bm{\mu} \Vert^2 +2\sigma^2\Vert \bm{\mu}\Vert^2 \vert \rho \vert$ yields
\begin{equation*}
    \exp\left(-\frac{z^2-2\Vert \bm{\mu} \Vert^2 z+2\sigma^2\Vert \bm{\mu}\Vert^2\vert\rho z\vert}{2\sigma^2\Vert \bm{\mu}\Vert^2}\right) \geq \exp\left(-\frac{z^2}{\sigma^2\Vert \bm{\mu}\Vert^2} \right).
\end{equation*}
Therefore, we have
\begin{equation*}
    4g''(\rho) \geq \frac{2}{\sigma\Vert \bm{\mu}\Vert\sqrt{2\pi }}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2} \right) \int_{2\Vert \bm{\mu} \Vert^2 +2\sigma^2\Vert \bm{\mu}\Vert^2 \vert \rho \vert}^{\infty} z^2\exp\left(-\frac{z^2}{\sigma^2\Vert \bm{\mu}\Vert^2} \right)dz.
\end{equation*}
The change of variables $z \mapsto \frac{\sigma\Vert\bm{\mu}\Vert}{\sqrt{2}} z$ yields
\begin{align*}
   g''(\rho) &\geq \frac{\sigma^2\Vert \bm{\mu}\Vert^2}{8\sqrt{\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}\right) \int_{ 2\sqrt{2}(1+2\vert \frac{\rho}{\rho^*}\vert)\frac{\Vert \bm{\mu} \Vert}{\sigma}}^{\infty} z^2\exp\left(-\frac{z^2}{2}\right)dz\\
   &\geq \frac{\Vert \bm{\mu} \Vert^4}{\sqrt{\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}\right) \int_{ 2\sqrt{2}(1+2\vert \frac{\rho}{\rho^*}\vert)\frac{\Vert \bm{\mu} \Vert}{\sigma}}^{\infty} \exp\left(-\frac{z^2}{2}\right)dz,
 \end{align*}
where the last inequality holds because $z^2 \ge \frac{8\norm{\bm{\mu}}^2}{\sigma^2}$ on the interval $[2\sqrt{2}(1+2 |\tfrac{\rho}{\rho^*}|) \tfrac{\norm{\bm{\mu}}}{\sigma}, \infty)$. Let $r(\rho) :=2\sqrt{2}(1+2\vert \frac{\rho}{\rho^*}\vert)$.
 Continuing, we get \begin{equation*}
     g''(\rho) \geq\frac{\Vert \bm{\mu} \Vert^4}{\sqrt{\pi}}\exp\left(-\frac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}\right) \Phi^{c}\left( \frac{r(\rho)\Vert \bm{\mu} \Vert}{\sigma}\right).
 \end{equation*}
 The proof now follows from Lemma \ref{lem:high_tech_not_appendix}.

\end{proof}
Let us recall that we define the target set $C$ by
\begin{equation}\label{eq:C_high_proof}
   C=\left\{\bm{\theta}: \vert \rho\sigma^2-2\vert < 1 \text{ and } \sigma \Vert \tilde{\bm{\theta}} \Vert \leq c \right\},
\end{equation}
where the constant $c$ is \begin{equation}\label{eq:c_high_proof}
    c=8\left(\frac{\tfrac{3}{2}\left(\frac{1}{0.16^2}+\sqrt{\frac{2}{\pi}}\frac{1}{0.16} \right)+1}{\Phi^c(1)}-\tfrac{1}{2}\log(2)\right) \approx 3404.46.
\end{equation}
The next proposition establishes the desired bound $\bE[\tau_m^C] \underset{\sim}{<} m.$ Note that the set $C$ is compact. This implies that the value of the test function at the next iteration increases, in expectation, by at most a fixed constant whenever the current iterate is inside $C$. This facilitates establishing bounds for $\bE[\tau_C^m]$ as we will in the proof of following proposition.
\begin{proposition}\label{prp:high_V} Let $\bm{\theta}_0=\bm{0}$ and suppose that $0.16\Vert \bm{\mu} \Vert \leq\sigma$. Consider the set $C$ defined in \eqref{eq:C_high_proof}.
Let the step-size $\alpha > 0$ satisfy
\begin{equation}\label{eq:high_noise_alpha_assumption}
\alpha < \frac{\min\{1,B\}}{\Vert \bm{\mu} \Vert^2+d\sigma^2},
\end{equation}
where the constant $B$ is
\begin{equation}
B:=\tfrac{\Vert \bm{\mu}\Vert^4}{4\sigma^4\sqrt{\pi}}\exp\left(-\tfrac{\Vert \bm{\mu} \Vert^2}{2\sigma^2}\right) \Phi^{c}\left( \tfrac{7\sqrt{2}\Vert \bm{\mu} \Vert}{\sigma}\right).
\end{equation}
Then the following is true
\begin{equation}\label{eq:high_Tm}
    \bE\left[\tau_m^{C}\right]\leq
    \frac{\Vert \bm{\mu} \Vert^2}{4\alpha\sigma^4\min\{ 1,B\}}+(m-1)\left(2+\frac{\Vert \bm{\mu} \Vert^2+c\sigma^2}{\alpha \sigma^4\min\{1,B\}}\right).
\end{equation}
\end{proposition}
\begin{proof}[Proof of Proposition \ref{prp:high_V}] To simplify the subsequent argument, we define the test function $\overline{V}$ by
\[
\overline{V}(\bm{\theta})=\frac{\Vert \bm{\theta}-\bm{\theta}^*\Vert^2}{\alpha\min\{1,B\}}.
\]
We first establish the drift equation \eqref{eq:st_VoutC} for the test function $\overline{V}$ and the target set \eqref{eq:C_high_proof}. To do so, we first show that
\begin{equation}\label{eq:high_thetak}
 f(\bm{\theta}_{k-1})-f(\bm{\theta}^*) \geq \min\{1,B\}1_{\{\bm{\theta}_{k-1}\not\in {C}\}}.
\end{equation}
Clearly \eqref{eq:high_thetak} holds when $\bm{\theta}_{k-1}\in {C}$ so suppose that $\bm{\theta}_{k-1}\not\in {C}$. Therefore, either we have $\vert\rho_{k-1}\sigma^2-2\vert>1$ or $\sigma \Vert \tilde{\bm{\theta}}_{k-1}\Vert \geq c$. If $\vert\rho_{k-1}\sigma^2-2\vert>1$, then Lemma \ref{lem: logistic_case_1} yields that $ f(\rho_{k-1}\bm{\mu})-f(\bm{\theta}^*)\geq B$. Then we write that
\begin{equation}
     f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)=\underbrace{f(\bm{\theta}_{k-1})-f(\rho_{k-1}\bm{\mu})}_{\geq 0 \text{ by Lemma~\ref{lem: logistic_case_2} }}+f(\rho_{k-1}\bm{\mu})-f(\bm{\theta}^*) \geq B,
\end{equation}
establishing \eqref{eq:high_thetak}. So assume $\vert \rho_{k-1} \sigma^2-2\vert<1$ and hence $\sigma \Vert \tilde{\bm{\theta}}_{k-1}\Vert \geq c$. We obtain that
\begin{align}
     \bE_{\bm{\xi}_k} \left[\log\left( \frac{1+\exp(-\bm{\theta}_{k-1}^T\bm{\xi}_k)}{1+\exp(-\rho_{k-1} \bm{\mu}^T\bm{\xi}_k)}\right)\right]&\geq \left ( \tfrac{1}{2} \log(2) + \tfrac{\sigma \Vert \tilde{\bm{\theta}}\Vert}{8} \right )\Phi^c(1)-\left | \frac{\rho_{k-1}}{4}\right | \left(\Vert \bm{\mu}\Vert^2+\sigma \Vert \bm{\mu} \Vert \sqrt{\frac{2}{\pi}} \right)\nonumber\\&\geq \left ( \tfrac{1}{2} \log(2) + \tfrac{\sigma \Vert \tilde{\bm{\theta}}\Vert}{8} \right )\Phi^c(1) -\frac{3}{2}\left(\frac{1}{0.16^2}+\frac{1}{0.16} \sqrt{\frac{2}{\pi}} \right)\geq 1\label{eq:high_noise_lastprop_101},
\end{align}
where the first inequality follows from  Lemma~\ref{lem: logistic_case_2} (Note: $\bm{\xi}_k$ and $\bm{\theta}_{k-1}$ are independent). In the second inequality, we used that $\rho_{k-1}<\tfrac{3}{\sigma^2}$ and also the assumption that $0.16\Vert \bm{\mu} \Vert \leq \sigma$. The last inequality follows from the assumption that $\sigma \Vert\tilde{\bm{\theta}}_{k-1}\Vert\geq c$.  We therefore have
\[
f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)=\underbrace{f(\bm{\theta}_{k-1})-f(\rho_{k-1}\bm{\mu})}_{\geq 1 \text{ by \eqref{eq:high_noise_lastprop_101}}}+\underbrace{f(\rho_{k-1}\bm{\mu})-f(\bm{\theta}^*)}_{\geq 0 \text{ by Lemma~\ref{lem: logistic_case_1}}} \geq 1.
\]
We have thus shown \eqref{eq:high_thetak}. By Lemma~\ref{lem:technical_convex_bound}, we have
  \begin{align}
       f(\bm{\theta}_{k-1})-f(\bm{\theta}^*)&\leq \tfrac{1}{2\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta}^* \Vert^2-\bE\left[\Vert \bm{\theta}_{k}-\bm{\theta}^* \Vert^2\, |\mathcal{F}_{k-1}\right] \right)+\tfrac{\alpha}{2}\left (\norm{\bm{\mu}}^2 + d \sigma^2 \right ) \nonumber\\ &\leq \tfrac{1}{2\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta}^* \Vert^2-\bE\left[\Vert \bm{\theta}_{k}-\bm{\theta}^* \Vert^2\, |\mathcal{F}_{k-1}\right] \right)+\tfrac{1}{2}\min\{1,B\}, \label{eq:high_decrease_v2}
    \end{align}
    where in the last inequality we used the assumption on $\alpha$, namely \eqref{eq:high_noise_alpha_assumption}. By \eqref{eq:high_thetak}, we obtain that
\[
\min\{1,B\}\left(2\cdot 1_{\{\bm{\theta}_{k-1}\notin C\}}-1\right)\leq \tfrac{1}{\alpha}\left(\Vert  \bm{\theta}_{k-1}-\bm{\theta}^* \Vert^2-\bE\left(\Vert \bm{\theta}_{k}-\bm{\theta}^* \Vert^2\, |\mathcal{F}_{k-1}\right] \right).
\]
A simple manipulation then yields
\begin{align}
    \bE\left[\overline{V}(\bm{\theta}_k)|\mathcal{F}_{k-1}\right] &\leq \overline{V}(\bm{\theta}_{k-1})-2\cdot 1_{\{\bm{\theta}_{k-1}\not\in {C} \}}+1 \nonumber \\&=\overline{V}(\bm{\theta}_{k-1})-1+2\cdot 1_{\{\bm{\theta}_{k-1}\in {C} \}}. \label{eq:high_noise_drift_eq}
\end{align}
From Appendix Lemma \ref{lem:drift_lemma_appendix}, we thus obtain that
\begin{equation}\label{eq:high_T_v2}
    \bE\left[\tau_m^{C}\right]\leq \overline{V}(\bm{0})+(m-1)\left(2+\sup_{\bm{\theta}\in {C}}\overline{V}(\bm{\theta})\right).
\end{equation}
 It remains to upper bound the quantity $\sup_{\bm{\theta}\in {c}}\overline{V}(\bm{\theta})$.  For any $\bm{\theta} \in C$, it holds that $\vert \rho\sigma^2-2\vert<1$. From this, we obtain that
\begin{align*}
    \Vert \bm{\theta}-\bm{\theta}^*\Vert^2&=\left( \rho \sigma^2-\rho^*\sigma^2 \right )^2\frac{\Vert \bm{\mu} \Vert^2}{\sigma^4}+\Vert \tilde{\bm{\theta}}\Vert^2 \\&\leq \frac{\Vert \bm{\mu} \Vert^2}{\sigma^4}+\frac{c}{\sigma^2},
\end{align*}
We, therefore, have
\begin{equation}
    \sup_{\bm{\theta}\in {C}}\overline{V}(\bm{\theta})\leq \frac{\Vert \bm{\mu} \Vert^2+c\sigma^2}{\alpha \sigma^4\min\{1,B\}}.
\end{equation}
The proof is complete.
\end{proof}
We are now ready to prove Theorem~\ref{thm.b}.
\begin{proof}[Proof of Theorem \ref{thm.b}]
It follows from Proposition~\ref{prp:high_V} that there exists some constant $r>0$ such that $\bE\left[\tau_m^{C}\right]\leq rm$ for all $m\geq 1$. Taken Lemma~\ref{lem:st_ETC} and Lemma~\ref{lem: high_noise_delta} together, we obtain that
\[
\bE[T]\leq \bE[T_C]\leq \sum_{m=1}^{\infty}\bE[\tau_m^{C}]\Phi^{c}\left(\frac{-\sigma}{\Vert \bm{\mu} \Vert}\right)^{m-1}.
\]
Therefore,
\[
\bE[T]\leq \bE[T_C]\leq r\sum_{m=1}^{\infty}m\Phi^{c}\left(\frac{-\sigma}{\Vert \bm{\mu} \Vert}\right)^{m-1}<+\infty.
\]
The proof is complete.
\end{proof}

Optimization and Control (math.OC); Machine Learning (stat.ML)