Reconstructing Point Sets from Distance Distributions

Abstract: We address the problem of reconstructing a set of points on a line or a loop from their unassigned noisy pairwise distances. When the points lie on a line, the problem is known as the turnpike; when they are on a loop, it is known as the beltway. We approximate the problem by discretizing the domain and representing the $N$ points via an $N$-hot encoding, which is a density supported on the discretized domain. We show how the distance distribution is then simply a collection of quadratic functionals of this density and propose to recover the point locations so that the estimated distance distribution matches the measured distance distribution. This can be cast as a constrained nonconvex optimization problem which we solve using projected gradient descent with a suitable spectral initializer. We derive conditions under which the proposed distance distribution matching approach locally converges to a global optimizer at a linear rate. Compared to the conventional backtracking approach, our method jointly reconstructs all the point locations and is robust to noise in the measurements. We substantiate these claims with state-of-the-art performance across a number of numerical experiments. Our method is the first practical approach to solve the large-scale noisy beltway problem where the points lie on a loop.

\subsection{Related Work}

In the noiseless case, Lemke and Werman \cite{Lemke:1988} addressed the turnpike problem via polynomial factorization. Namely, the polynomial $Q_\mathcal{D}(a) = N+\sum_{k=1}^K(a^{d_k}+a^{-d_k})$ is invariant to permutations of pairwise distances. If one can factorize it as $Q_\mathcal{D}(a)=R(a)R(a^{-1})$ where $R(a)=\sum_{n=1}^N a^{u_n}$, then the point locations can be read off from the exponents. When the distances are all integers, the factorization runs in a time that is polynomial in the degree of $Q(a)$ \cite{Lenstra1982}, which is the largest pairwise distance. However, this approach quickly becomes impractical, and is brittle in the presence of noise.

The more practical backtracking algorithm by Skiena et al. \cite{Skiena:1990} produces a solution for typical instances in time $\mathcal{O}(n^2\log n)$. It progressively finds the assignment for the remaining largest unassigned distance in $\mathcal{D}$, and adopts the branch-and-bound search strategy to recover the point locations in a depth-first manner. However, there exist examples with exponential runtime \cite{Skiena:1990,ZhangExp:1994}. Abbas and Bahig \cite{Abbas2016} later demonstrated that some of the worst-case scenarios could be avoided by performing a breadth-first search instead. An alternative to clever combinatorial search is to formulate the problem as a binary integer program \cite{Miller:1960,IBARAKI197639,Papadimitriou:1982}, and then relax it to obtain a convex semidefinite program \cite{Dakic:2000}. One drawback of this scheme is that it is computationally infeasible for large-scale problems. In this paper we propose to relax the integer program to a constrained nonconvex optimization problem that can be solved efficiently using projected gradient descent with a spectral initializer.

To address the noisy case where the turnpike problem becomes NP-hard \cite{Cieliebak:2004}, Skiena and Sundaram proposed a modification of the backtracking algorithm where an interval is associated with each recovered point to account for the uncertainty \cite{Skiena:1994}. As a consequence, the number of backtracking paths could grow exponentially large. Pruning can be performed on the paths when the relative errors in the distances are small; however, it requires careful adaptive tuning and could lead to no solution sometimes. Our approach naturally incorporates noise into the problem formulation, thus exhibiting better performance compared to the current state-of-the-art backtracking approach.

We mention that the turnpike problem is also related to the problem of string reconstruction from substring compositions which arises in protein mass spectrometry \cite{Acharya2015StringRF,Bulteau:2014,LEE2013}. The advances presented here for the turnpike problem might inspire similar approaches to solve its string variant.

The beltway problem is more difficult than the turnpike problem \cite{Skiena:1990, Lemke2003}. Due to the loop structure, it can no longer be formulated as a polynomial factorization problem. It is also impossible for the backtracking approach to rely on the remaining largest unassigned distance to find the point locations progressively \cite{Lemke2003}. For small problems, Fomin \cite{Fomin:2016:1,Fomin:2016:2} proposed to avoid an exhaustive search in the noiseless case by further removing the redundant distances from $\mathcal{H}$ sequentially, and later extended it to handle noisy measurements\cite{Fomin:2019:3}. To the best of our knowledge, our work in this paper offers an alternative by providing the first practical approach to solve the large-scale noisy beltway problem.

\subsection{Uniqueness}

One complication with the turnpike problem is that the solution is not necessarily unique (up to a relabeling of the points and up to a congruence). Fortunately, the solution to the uDGP in any dimension is known to be \textit{generically} unique, in the sense made precise in the form of the reconstructability tests for the point configurations by Boutin and Kemper in \cite[Theorem 2.6 and Proposition 2.11]{BOUTIN2004709}. For example, if the points are sampled iid from an absolutely continuous probability distribution, then almost surely the distance distribution specifies their geometry uniquely (up to a relabeling and a congruence).

Boutin and Kemper worked with complete distance measurements. Gortler et al. \cite{GUGR2018} later relaxed the completeness assumption and only required the underlying graph to be generically globally rigid \cite{CGGR2010}. Under this sufficient condition, they proved that the reconstruction of a generic point configuration is unique.

Importantly, beyond uniqueness, Boutin and Kemper \cite{BOUTIN2004709} showed that when the multiset $\mathcal{D}$ in the turnpike problem contains only distinct distances, there is a suitably defined neighborhood around each uniquely reconstructable point configuration such that all configurations within the neighborhood are also uniquely reconstructable, and the forward and backward mappings between the different distance multisets are continuous. To the best of our knowledge, there has not been much work on the uniqueness of beltway reconstructions. In the remainder of this paper, we will assume that the measured distances correspond to a uniquely reconstructable configuration.

\subsection{Our Approach}

The combinatorial turnpike problem can be formulated as an assignment problem \cite{Assignment:1957,Burkard:2009} or a general integer program (when the domain is discrete) \cite{Miller:1960,IBARAKI197639,Papadimitriou:1982}. Most of the prior approaches described in the above sections try to first find the correct assignments of the distances to pairs of points $\alpha(k)$, and then recover the point locations $u_n$. On the other hand, the approach by Daki\'c \cite{Dakic:2000} adopts the integer programming formulation where the point locations are represented by a binary vector in the noiseless case, and directly recovered via semidefinte programming (SDP). Assignments are then a byproduct of the process.

We proceed along the line of integer programming to solve the turnpike problem and the beltway problem in section \ref{sec:main_turnpike} and \ref{sec:main_beltway} respectively. Instead of relaxing the integer program to a convex SDP, we relax it to a constrained minimization of a nonconvex objective, which leads to the proposed approach that is more efficient and suitable for large-scale problems. Importantly, measurement noise is naturally incorporated into our formulation by smoothing the target distance distribution. A key ingredient in the proposed approach is a suitably constructed initializer inspired by the spectral initialization strategy \cite{Netrapalli2015:RPAM,WF:2015}. We analyze the convergence of the projected gradient method to a global optimum. In order to have a fast method, we also propose a computationally efficient projection onto the relaxed constraint set.

Both the turnpike and beltway problems can be formulated in similar ways. Starting with the easier turnpike problem, we shall present the proposed approach in detail in section \ref{sec:main_turnpike}, and then demonstrate how it can be adapted to solve the beltway problem as well in section \ref{sec:main_beltway}. Numerical experiments in section \ref{sec:exp} show that our method achieves state-of-the-art performances for the turnpike recovery, and is the first practical approach to solve the large-scale noisy beltway problem. We conclude this paper with a discussion of our results in Section \ref{sec:con}. The proofs of the formal results can be found in the Appendix.
\subsection{Distance Distribution Matching}
\label{subsec:ddm}

\begin{figure*}[tbp]
\centering
\subfigure[]{
\label{fig:prob_approx_cont}
\includegraphics[width=2.5in]{figures/prob_gaussian_convolve}}
\subfigure[]{
\label{fig:prob_approx_disc}
\includegraphics[width=2.5in]{figures/prob_gaussian_convolve_histogram}}
\caption{(a) The approximated distribution $p(d)$ based on the distance multiset $\mathcal{D}$; (b) The discretized distance distribution $p(y)$ from $p(d)$.}
\label{fig:prob_dist_discretization}
\end{figure*}

Depending on how the distance $d_k$ is measured in various applications, a variety of noise models for $w_k$ may be appropriate \cite{Oppenheim:2009,Vaseghi:2006}. Here we model $w_k$ as iid zero-mean Gaussian noise with variance $\xi^2$
\[w_k\sim\mathcal{N}(0,\xi^2)=\frac{1}{\sqrt{2\pi\xi^2}}\exp\left(-\frac{1}{2\xi^2}w_k^2\right)\,.\]
Ideally, we would like to find a set of point locations so that the \emph{estimated} distance distribution matches the \emph{oracle} distance distribution $g(d)$
\begin{align}
\label{eq:oracle_dist}
g(d)=\frac{1}{K}\cdot\sum_{k=1}^{K}\mathcal{N}\left(d\ \left|\ s_k,\xi^2\right.\right) :=\frac{1}{K}\sum_{k=1}^K\frac{1}{2\pi\xi^2}\exp\left(-\frac{(d-s_k)^2}{2\xi^2}\right)\,.
\end{align}
Since $g(d)$ is in general unknown, we approximate it here using the following distribution $p(d)$ based on the measured distances in $\mathcal{D}$.
\begin{align}
\label{eq:approx_dist}
p(d)=\frac{1}{K}\cdot\sum_{k=1}^{K}\mathcal{N}\left(d\ \left|\ d_k, \sigma^2\right.\right)\,,
\end{align}
where $\sigma^2$ should be tuned according to an a priori estimate of the level of noise in the data and the grid resolution. As shown in Fig. \ref{fig:prob_dist_discretization}, the distribution $p(d)$ is further discretized to $p(y)$ in order to perform distribution matching with respect to the quantized distance $y$.
\begin{align}
\label{eq:obs_dist}
p(y)=\int_{(y-0.5)\Delta l}^{(y+0.5)\Delta l}p(d)\ \textnormal{d}d\,.
\end{align}
Similar to \eqref{eq:quad_form_p}, the estimated distribution $q(y)$ can also be expressed in terms of the solution ${\bm{z}}$
\begin{align}
\label{eq:quad_form_q}
q(y)=\frac{1}{K}\cdot{\bm{z}}^T{\bm{A}}_y{\bm{z}}\,,
\end{align}
where $z_m$ is the estimated (unnormalized) probability that a point is located at $l_m$. We can solve for it by minimizing the mean-squared error between $q(y)$ and $p(y)$ subject to the constraints in \eqref{eq:relaxed_constraint} and \eqref{eq:l1_constraint}:
\begin{align}
\label{eq:constrained_nonconvex}
\begin{split}
\min_{{\bm{z}}}\quad &f({\bm{z}})=\frac{1}{M}\sum_{y=0}^{M-1}\big(q(y)-p(y)\big)^2
\end{split}\\
\label{eq:relaxed_constraint}
\textnormal{subject to}\quad &0\leq z_m\leq 1,\ \forall\ m\in\set{1,\cdots,M}\\
\label{eq:l1_constraint}
&\|{\bm{z}}\|_1 = N\,.
\end{align}

\subsection{Extracting Point Locations from the Estimated Distribution}

In general, the recovered vector ${\bm{z}}$ will not be supported on exactly $N$ indices. In the following we discuss how to extract the $N$ point location estimates when this is the case.

\paragraph{Noiseless case.} If we assume that there is no measurement noise in $d_k$ and no quantization error in the quantized distance $y_k=\left\lfloor\frac{d_k}{\Delta l}\right\rceil$, the vector ${\bm{x}}$ is then binary: ${\bm{x}}\in\set{0,1}^M$. Suppose ${\bm{z}}^\dagger$ is one of the global optimizers of \eqref{eq:constrained_nonconvex} that is different from ${\bm{x}}$ and $f({\bm{z}}^\dagger)=0$. We then have from $q(y=0)=p(y=0)$ that
\[
\|{\bm{z}}^\dagger\|_2^2={{\bm{z}}^\dagger}^T{\bm{A}}_0{\bm{z}}^\dagger = {\bm{x}}^T{\bm{A}}_0{\bm{x}}=\|{\bm{x}}\|_2^2=N\,.
\]
From \eqref{eq:l1_constraint}, we can get
\begin{align}
\label{eq:noiseless_global_opt}
\|{\bm{z}}^\dagger\|_2^2=\|{\bm{z}}^\dagger\|_1=N\,.
\end{align}
If the $m$-th entry $z^\dagger_m\in(0,1)$, then $\|{\bm{z}}^\dagger\|_2^2<\|{\bm{z}}^\dagger\|_1$ which is in contradiction with \eqref{eq:noiseless_global_opt}. Hence $z^\dagger_m\notin(0,1)$, and the global optimizer is integer-valued, ${\bm{z}}^\dagger\in\set{0,1}^M$. The points are estimated at the segments that correspond to the $1$-entries in ${\bm{z}}^\dagger$.

If the solution ${\bm{z}}$ is not a global optimizer, then ${\bm{z}}\in[0,1]^M$. The point locations can be extracted in the same way as in the noisy case which we describe next.

\begin{figure*}[tbp]
\centering
\subfigure{
\label{fig:z_vec}
\includegraphics[width=\textwidth]{figures/z_vec.png}}\\
\subfigure{
\label{fig:h_clustering}
\includegraphics[width=\textwidth]{figures/h_clustering.png}}
\caption{Illustration of agglomerative clustering for $N=5$. The agglomerative clustering produces $8$ clusters, only the centroids of the $5$ clusters with the highest weights are taken as the point locations.}
\label{fig:hierarchical_clustering}
\end{figure*}

\paragraph{Noisy case.} In the noisy case we have ${\bm{x}}\in[0,1]^M$. The $m$-th entry $z_m$ of ${\bm{z}}$ is the \emph{estimated} probability that a point is located at the $m$-th segment $l_m$.
Extracting $N$ point locations from ${\bm{z}}$ can be posed as a clustering problem. As illustrated in Fig. \ref{fig:hierarchical_clustering}, each $l_m$ is viewed as a cluster with the weight $z_m$. We can cluster the $M$ segments using the agglomerative clustering approach \cite{Rokach2005} summarized in Algorithm \ref{alg:agg}. The centroids of the $N$ clusters with the largest weights are taken as the estimated point locations.
\begin{algorithm}[tbp]
\caption{Extracting the point locations via agglomerative clustering }
\label{alg:agg}
\begin{algorithmic}[1]
\REQUIRE The solution ${\bm{z}}$, the smallest distance between two different points $d_{\min}$.
\STATE Treat each segment $l_m$ with a nonzero weight $\omega_m=z_m$ as one cluster $C_m=\set{l_m}$
\STATE Compute the centroid $c_m$ of every cluster $C_m\in\mathcal{C}=\set{C_1,C_2,\cdots}$
\WHILE{$|\mathcal{C}|>N$}
\STATE Merge the two closest clusters\footnotemark $\set{C_i,C_j}$ with weights $\set{w_i<1,w_j<1}$ and centroids $\|c_i-c_j\|<d_{\min}$ into one cluster $C_i$
\STATE Update the weight $w_i$ and the centroid $c_i$ of the new cluster $C_i$
\IF{the clusters cannot be merged further}
\STATE \textbf{break}
\ENDIF
\ENDWHILE
\STATE {\bfseries Return} the set of centroids $\set{c_1,c_2,\cdots}$
\end{algorithmic}
\end{algorithm}
\footnotetext{Randomly pick a pair of clusters in case of a draw.}

\subsection{Projected Gradient Descent}
Let $\mathcal{S}$ denote the convex set defined by the constraints \eqref{eq:relaxed_constraint},\eqref{eq:l1_constraint}. Given a proper initialization ${\bm{z}}_0$, we propose to solve \eqref{eq:constrained_nonconvex} via the projected gradient descent method:
\begin{align}
\label{eq:pgd_update}
{\bm{z}}_{t+1} = \mathscr{P}_\mathcal{S}\big({\bm{z}}_t-\eta\cdot\nabla f({\bm{z}}_t)\big)\,,
\end{align}
where $\eta>0$ is the step size, $\mathscr{P}_\mathcal{S}(\cdot)$ is the projection of the gradient descent update onto $\mathcal{S}$, and $\nabla f({\bm{z}}_t)$ is the gradient
\begin{align}
\nabla f({\bm{z}}_t) = \frac{2}{MK^2}\sum_{y=0}^{M-1}\left({\bm{z}}_t^T{\bm{A}}_y{\bm{z}}_t-{\bm{x}}^T{\bm{A}}_y{\bm{x}}\right)\cdot\left({\bm{A}}_y+{\bm{A}}_y^T\right){\bm{z}}_t\,,
\end{align}
where both $q(y)$ and $p(y)$ are replaced with their quadratic forms in \eqref{eq:quad_form_p} and \eqref{eq:quad_form_q}. An adaptive strategy can be used to determine some suitable step size $\eta>0$ to minimize the objective function. The proposed approach is finally summarized by Algorithm \ref{alg:pdp_pgd}.
\begin{algorithm}[tbp]
\caption{The noisy turnpike problem via projected gradient descent}
\label{alg:pdp_pgd}
\begin{algorithmic}[1]
\REQUIRE the distance multiset $\mathcal{D}$, the number of points $N$, quantization step size $\Delta l$, gradient descent step size $\eta$, adaptive rate $\beta\in(0,1)$, convergence threshold $\epsilon$, the maximum number of iterations $T$.
\STATE Compute the discrete approximated distribution $q(y)$ from $\mathcal{D}$
\STATE Compute the spectral initializer ${\bm{z}}_0$
\FOR{$t=\{0,1,\cdots,T\}$}
    \WHILE{true}
    \STATE Compute the projected gradient descent update ${\bm{z}}_{t+1}=\mathscr{P}_\mathcal{S}\big({\bm{z}}_t-\eta\cdot\nabla f({\bm{z}}_t)\big)$
    \IF {$f({\bm{z}}_{t+1})\leq f({\bm{z}}_t)$}
        \STATE Increase the step size $\eta=\frac{1}{\beta}\cdot\eta$
        \STATE \textbf{break}
    \ELSE
        \STATE Decrease the step size $\eta=\beta\cdot\eta$
    \ENDIF
    \ENDWHILE
    \IF {$\frac{\|{\bm{z}}_{t+1}-{\bm{z}}_t\|_2}{\|{\bm{z}}_t\|_2}<\epsilon$}
        \STATE Convergence is reached, set ${\bm{z}}={\bm{z}}_{t+1}$
        \STATE \textbf{break}
    \ENDIF
\ENDFOR

\STATE {\bfseries Return} ${\bm{z}}$
\end{algorithmic}
\end{algorithm}

\subsubsection{Spectral Initialization}
\label{subsec:spec_init}
\begin{figure*}[tbp]
\centering
\includegraphics[width=\textwidth]{figures/spec_init_1000}
\caption{An example of the obtained spectral initializer ${\bm{z}}_0$. The entries corresponding to the neighbourhood of the true point locations (illustrated by vertical lines) in general have larger values, indicating higher confidence in those locations.}
\label{fig:spec_init}
\end{figure*}

A suitable initialization is needed to solve the constrained noncovex problem in \eqref{eq:constrained_nonconvex} via projected gradient descent. Here we can borrow ideas from another problem with quadratic measurements, the phase retrieval problem \cite{Gerchberg:72,Fienup:82}. In phase retrieval, the task is to compute a complex signal ${\bm{v}}\in\mathbb{C}^M$ from its quadratic measurements of the form $\mu_i = |\langle{\bm{v}}, {\bm{a}}_i\rangle|^2$ for $1 \leq i \leq I$. Since $\mu_i = {\bm{v}}^* {\bm{a}}_i {\bm{a}}_i^* {\bm{v}}$, spectral initialization for phase retrieval is based on a weighted sum of the rank-1 measurement matrices ${\bm{a}}_i {\bm{a}}_i^*$. Namely, using matrix concentration results, Netrapalli et al. \cite{Netrapalli2015:RPAM} showed that the leading eigenvector of $\sum_{i=1}^I \mu_i {\bm{a}}_i {\bm{a}}_i^*$ is close to the the true ${\bm{v}}$. Similar arguments can be used for quadratic systems of full-rank random matrices \cite{QuadFR:2019}.

In our formulation of the turnpike problem \eqref{eq:quad_form_p}, the rank-1 matrices ${\bm{a}}_i {\bm{a}}_i^*$ are replaced by ${\bm{A}}_y$ which are not necessarily PSD nor rank-1; they are also deterministic. Notwithstanding, we can use the spectral initialization strategy. As we shall see from the numerical experiments in Section \ref{sec:exp}, this strategy works well empirically, although a rigorous proof remains an open question.

Let $\beta_y = \frac{p(y)\cdot K}{\|{\bm{A}}_y\|_F}$ and ${\bm{H}}_y=\frac{{\bm{A}}_y}{\|{\bm{A}}_y\|_F}$. We can rewrite \eqref{eq:quad_form_p} as
\begin{align}
\beta_y={\bm{x}}^T{\bm{H}}_y{\bm{x}}=\langle{\bm{H}}_y,\ {\bm{x}}\vx^T\rangle =: \langle {\bm{H}}_y,\ {\bm{X}}\rangle\,.
\end{align}
The set $\left\{{\bm{H}}_y,\ 0\leq y\leq M-1\right\}$ can be viewed as an orthonormal basis for the matrix subspace $\mathrm{span} \, \set{{\bm{H}}_1, \ldots {\bm{H}}_{M-1}}$,
\begin{align}
    \langle {\bm{H}}_i,\ {\bm{H}}_j \rangle=\left\{\begin{array}{l}
    1\\
    0
    \end{array}
    \quad
    \begin{array}{l}
    \textnormal{if }i=j\\
    \textnormal{if }i\neq j\,
    \end{array}.
    \right.
\end{align}
With this interpretation, $\beta_y$ becomes the expansion coefficient of ${\bm{X}}$ in the direction of ${\bm{H}}_y$. The least squares estimate of ${\bm{X}}$ is then
\begin{align}
    \widehat{{\bm{X}}}=\sum_{y=0}^{M-1}\beta_y\cdot{\bm{H}}_y\,,
\end{align}
which is nothing but the orthogonal projection of ${\bm{X}}$ on the subspace spanned by the ${\bm{H}}_y$. Finally, we find the spectral initializer ${\bm{z}}_0$ so that ${\bm{z}}_0 {\bm{z}}_0^T$ is close to $\widehat{{\bm{X}}}$ in Frobenius norm subject to the constraint that $\|{\bm{z}}_0\|^2_2=N$. Let the spectral initializer ${\bm{z}}_0=\sqrt{N}{\bm{e}}_{\max}$, where $\|{\bm{e}}_{\max}\|_2=1$. We have
\begin{align}
\begin{split}
{\bm{e}}_{\max}&=\mathrm{argmin}_{{\bm{e}}}\ \|\widehat{{\bm{X}}}-N{\bm{e}}\ve^T\|_F^2\\
&=\mathrm{argmin}_{{\bm{e}}}\ \|\widehat{{\bm{X}}}\|_F^2+N^2\|{\bm{e}}\ve^T\|_F^2-2N\langle\widehat{{\bm{X}}},{\bm{e}}\ve^T\rangle\\
&=\mathrm{argmax}_{{\bm{e}}}\ {\bm{e}}^T\widehat{{\bm{X}}}{\bm{e}}\,.
\end{split}
\end{align}
\begin{itemize}
\item When $\widehat{{\bm{X}}}$ is symmetric, ${\bm{e}}_{\max}$ is given by the leading singular vector of $\widehat{{\bm{X}}}$ that corresponds to the largest singular value.
\item When $\widehat{{\bm{X}}}$ is not symmetric, we use the method of Lagrange multipliers and find the stationary points of the Lagrangian $\mathcal{L}({\bm{e}},\lambda)={\bm{e}}^T\widehat{{\bm{X}}}{\bm{e}}-\lambda({\bm{e}}^T{\bm{e}}-1)$. Setting the gradients to $0$, we have
\begin{align}
    \left(\widehat{{\bm{X}}}+\widehat{{\bm{X}}}^T\right){\bm{e}}&=2\lambda{\bm{e}}\\
    {\bm{e}}^T{\bm{e}}&=1
\end{align}
The stationary points are given by the eigenvectors of $\widehat{{\bm{X}}}+\widehat{{\bm{X}}}^T$, with ${\bm{e}}_{\max}$ being the one that corresponds to the largest eigenvalue. In practice we find it via the power iteration.
\end{itemize}

\remove{
\begin{algorithm}[tbp]
\caption{Spectral initialization}
\label{alg:spec_init_max}
\begin{algorithmic}[1]
\REQUIRE the least square estimate $\widehat{{\bm{X}}}$.
\STATE Initialize ${\bm{e}}_0$ with all $1$s.
\FOR{$t=\{0,1,\cdots,T\}$}
    \STATE Compute ${\bm{e}}_{t+1}=\frac{\left(\widehat{{\bm{X}}}+\widehat{{\bm{X}}}^T\right){\bm{e}}_t}{\left\|\left(\widehat{{\bm{X}}}+\widehat{{\bm{X}}}^T\right){\bm{e}}_t\right\|_2}$
    \IF {Convergence is reached}
        \STATE Set ${\bm{z}}_0=\sqrt{{\bm{e}}_{t+1}^T\widehat{{\bm{X}}}{\bm{e}}_{t+1}}\cdot{\bm{e}}_{t+1}$
        \STATE \textbf{break}
    \ENDIF
\ENDFOR
\STATE {\bfseries Return} ${\bm{z}}_0$
\end{algorithmic}
\end{algorithm}
}

\subsubsection{Efficient Projection onto the $l_1$-ball with Box Constraints}
\begin{figure*}[tbp]
\centering
\includegraphics[height=2.5in]{figures/projection}
\caption{ The gradient descent update ${\bm{z}}={\bm{z}}_t-\eta \nabla f({\bm{z}}_t)$ is projected back to the convex set $\mathcal{S}$.}
\label{fig:projection}
\end{figure*}

As shown in Fig. \ref{fig:projection}, the gradient descent update ${\bm{z}}={\bm{z}}_t-\eta\cdot\nabla f({\bm{z}}_t)$ is projected back onto the convex set $\mathcal{S}$, which is the $l_1$-ball with box constrains defined by \eqref{eq:relaxed_constraint} and \eqref{eq:l1_constraint}. The projection is the solution to the following convex problem
\begin{align}
\label{eq:projection_box_constrains}
\begin{split}
\min_{\bm{s}}\quad&\frac{1}{2}\|{\bm{s}}-{\bm{z}}\|_2^2\\
\textnormal{subject to}\quad&0\leq s_m\leq 1,\ \forall\ m\in[M]\\
&\|{\bm{s}}\|_1=N\,.
\end{split}
\end{align}
Duchi et al. \cite{projection06,Duchi:2008} proposed an efficient algorithm to compute the projection onto the $l_1$-ball when $s_m$ is only lower-bounded by $0$. Gupta et al. \cite{l1_box10, l1_box12} later extended that approach to handle projections with box constraints, when $s_m$ is both lower-bounded and upper-bounded. However, their approach is based on a sequential search for an optimal threshold $\kappa$, which is inefficient and cannot be parallelized for large-scale problems. Building on the work of \cite{projection06}, we address these issues by deriving a closed-form expression for the optimal $\kappa$ in \eqref{eq:kappa_compute} in terms of the entry index $r$ of a sorted ${\bm{s}}$.

Specifically, the Lagrangian of \eqref{eq:projection_box_constrains} is:
\begin{align}
\mathcal{L}=\frac{1}{2}\|{\bm{s}}-{\bm{z}}\|_2^2+\kappa\left(\|{\bm{s}}\|_1-N\right)-{\boldsymbol\zeta}^T\cdot{\bm{s}}+{\boldsymbol\xi}^T\cdot({\bm{s}}-\boldsymbol 1)\,,
\end{align}
where $\kappa\in\mathbb{R}$ is a real Lagrange multiplier, ${\boldsymbol\zeta}\in\mathbb{R}_+^M$, ${\boldsymbol\xi}\in\mathbb{R}_+^M$ are the nonnegative Lagrange multipliers. Taking the subgradient of $\mathcal{L}$ w.r.t. ${\bm{s}}$, and setting it to $0$, we have
\begin{align}
\frac{\partial\mathcal{L}}{\partial s_m}=s_m-z_m+\kappa-\zeta_m+\xi_m=0\,.
\end{align}
Since $\mathcal{S}$ is a closed convex set, the projection solution ${\bm{s}}$ exists and is unique. We need to consider the following two cases.
\begin{enumerate}[label={\arabic*)}]
\item If the solution ${\bm{s}}$ contains only $[0,1]$-entries, there are $N$ entries in ${\bm{s}}$ that equal $1$, and their indices correspond to the top $N$ entries of ${\bm{z}}$.
\item If at least one entry of ${\bm{s}}$ is between $0$ and $1$, the complementary slackness KKT condition indicates that when $0< s_m < 1$, the Lagrange multipliers $\zeta_m=\xi_m=0$. We then have:
\begin{align}
\label{eq:project_one}
s_m=z_m-\kappa\quad\quad\textnormal{if } 0< s_m < 1\,.
\end{align}
The above \eqref{eq:project_one} gives us an efficient way to compute $s_m$ if it happens to be between $0$ and $1$: simply subtract the threshold $\kappa$ from $z_m$. In order to find the optimal solution ${\bm{s}}$, we need to compute $\kappa$ and identify the three types of entries of ${\bm{s}}$: those that equal $0$, those that equal $1$, and those that are between $0$ and $1$. We will make use of the following lemma from \cite{projection06} about the entries of ${\bm{s}}$ that equal $0$:
\begin{lemma}
\label{lemma:lower_bound}
\cite {projection06} Let ${\bm{s}}$ be the optimal solution to the minimization problem in (\ref{eq:projection_box_constrains}). Let $i$ and $j$ be two indices such that $z_i>z_j$. If $s_i=0$ then $s_j$ must be $0$ as well.
\end{lemma}
Similarly, we can prove the following lemma about the entries of ${\bm{s}}$ that equal $1$ (the proof can be found in Appendix \ref{proof:lemma:upper_bound}).
\begin{lemma}
\label{lemma:upper_bound}
Let ${\bm{s}}$ be the optimal solution to the minimization problem in (\ref{eq:projection_box_constrains}). Let $i$ and $j$ be two indices such that $z_i>z_j$. If $s_j=1$ then $s_i$ must be $1$ as well.
\end{lemma}

Since reordering of the entries of ${\bm{z}}$ does not change the value of (\ref{eq:projection_box_constrains}), and adding some constant to ${\bm{z}}$ does not change the solution of (\ref{eq:projection_box_constrains}), without loss of generality we can assume that the entries of ${\bm{z}}$ are all positive in a non-increasing order: $z_1\geq z_2\geq\cdots\geq z_M\geq N$. Lemma \ref{lemma:lower_bound} and \ref{lemma:upper_bound} imply that for the optimal solution ${\bm{s}}$:
\begin{itemize}
\item The entries of ${\bm{s}}$ are in a non-increasing order.
\item The first $\rho$ entries of ${\bm{s}}$ satisfy $0<s_m\leq 1$; the rest of the entries are $0$s.
\end{itemize}
Since $\exists\ s_m\in(0,1)$, we have $\rho>N$ and that at most $N-1$ entries of ${\bm{s}}$ could equal $1$. Suppose the first $r-1$ entries of ${\bm{s}}$ are all $1$s, the following must hold for $1\leq r\leq N<\rho$
\begin{align}
\label{eq:cst_r}
&0<z_r-\kappa< 1\\
\label{eq:cst_rm1}
&1\leq z_{r-1}-\kappa,\quad\textnormal{if }2\leq r\leq N<\rho\,.
\end{align}

\begin{algorithm}[tbp]
\caption{Efficient projection onto the $l_1$-ball with box constraints}
\label{alg:ep_box}
\begin{algorithmic}[1]
\STATE Shift ${\bm{z}}$ s.t. $z_m\geq N$, $\forall\ m\in\set{1,\cdots,M}$; and sort ${\bm{z}}$ in a non-increasing order.
\FOR{$r=1:N$}
    \STATE Construct ${\bm{v}}$ out of ${\bm{z}}$ by removing the first $r-1$ entries, and compute $\rho_v$ according to \cite{Duchi:2008}:\\
           $\rho_v=\max\left\{l\in[N-r+1]\,:\, v_l-\textstyle\frac{1}{l}\left(\textstyle\sum_{m=1}^lv_m-(N-r+1)\right)>0\right\} $
    \STATE Compute $\kappa_v=\frac{1}{\rho_v}\left(\sum_{m=1}^{\rho_v}v_m-(N-r+1)\right)$
    \STATE Check if $(\rho_v, \kappa_v)$ satisfy \eqref{eq:cst_r} by examining $\widehat{s}_r=z_r-\kappa_v$
    \IF{$0<\widehat{s}_r< 1$}
        \IF{$r=1$}
            \STATE Set $\kappa=\kappa_v$, $\rho=\rho_v+r-1$ and \textbf{break}
        \ELSE
            \STATE Check if $(\rho_v, \kappa_v)$ satisfy \eqref{eq:cst_rm1} by examining $\widehat{s}_{r-1}=z_{r-1}-\kappa_v$
            \IF{$\widehat{s}_{r-1}\geq 1$}
                \STATE Set $\kappa=\kappa_v$, $\rho=\rho_v+r-1$ and \textbf{break}
            \ENDIF
        \ENDIF
    \ELSE
        \STATE \textbf{continue}
    \ENDIF
\ENDFOR
\IF{$(r,\rho,\kappa)$ can be found}
\STATE Compute ${\bm{s}}=\max\{{\bm{z}}-\kappa, 0\}$ followed by ${\bm{s}}=\min\{{\bm{s}},1\}$
\ELSE
\STATE Compute ${\bm{s}}$ by setting the top $N$ entries of ${\bm{z}}$ to $1$ and the rest entries to $0$
\ENDIF
\STATE {\bfseries Return} ${\bm{s}}$
\end{algorithmic}
\end{algorithm}

We can write the sum of ${\bm{s}}$ as follows:
\begin{align}
\sum_{m=1}^Ms_m=\sum_{m=1}^\rho s_m=(r-1)+\sum_{m=r}^\rho (z_m-\kappa)=N\,.
\end{align}
We then have:
\begin{align}
\label{eq:kappa_compute}
\kappa=\frac{1}{\rho-r+1}\left(\sum_{m=r}^\rho z_m-(N-r+1)\right)\,.
\end{align}
Finally, we can write the minimizing ${\bm{s}}$ as
\begin{align}
\label{eq:min_s}
{\bm{s}}=\left\{
\begin{array}{l}
1, \\
z_m-\kappa, \\
0,
\end{array} \quad
\begin{array}{l}
\textnormal{if }m\leq r-1\\
\textnormal{if }r\leq m\leq \rho\\
\textnormal{if }\rho+1\leq m\leq M\,.
\end{array}
\right.
\end{align}

If $r$ is known, we can find the value of $\rho$ efficiently using the approach in \cite{projection06,Duchi:2008}, and thus identify the three types of entries in ${\bm{s}}$. The threshold $\kappa$ and the solution ${\bm{s}}$ can be then computed using \eqref{eq:kappa_compute} and \eqref{eq:min_s} respectively. According to the following lemma (proved in Appendix \ref{proof:lemma:unique_r}), we can find $r$ by checking the integers in the set $\set{1,\ldots,N}$ one by one or in parallel until the computed $(\rho, \kappa)$ satisfy the two constraints \eqref{eq:cst_r} and \eqref{eq:cst_rm1}.
\begin{lemma}
\label{LEMMA:UNIQUE_R}
If the solution ${\bm{s}}$ has at least one entry $s_m\in(0,1)$, there is one and only one $r\in\set{1,\ldots,N}$ that produces the $(\rho,\kappa)$ satisfying \eqref{eq:cst_r} and \eqref{eq:cst_rm1}.
\end{lemma}
\end{enumerate}
In practice we do not know beforehand what the solution ${\bm{s}}$ is like. Given the uniqueness of the solution, we could start by trying to look for the right $(r,\rho,\kappa)$-values. If they can be found, ${\bm{s}}$ can then be computed using \eqref{eq:min_s}. If they can not be found, it means that ${\bm{s}}$ contains only $[0,1]$-entries and can be obtained straightforwardly. The proposed approach to perform efficient projection on the $l_1$-ball with box constraints is summarized in Algorithm \ref{alg:ep_box}.

\subsection{Convergence Analysis}
\label{subsec:cvg}
In this section we study the convergence behavior of the proposed approach in the neighbourhood $\mathcal{E}(\tau)$ around a global optimizer ${\bm{x}}$.
\[\mathcal{E}(\tau)=\set{{\bm{z}}\ |\ \|{\bm{z}}-{\bm{x}}\|_2<\tau\,,\ {\bm{z}}\in\mathcal{S}}\,,\]

\paragraph{Noiseless recovery.} In this case we have ${\bm{x}}\in\set{0,1}^M$. There exists some $\tau>0$ that depends on ${\bm{x}}$, such that if the $t$-th iterate ${\bm{z}}_t\in\mathcal{E}(\tau)$, the projected gradient descent update in \eqref{eq:pgd_update} is guaranteed to converge linearly to ${\bm{x}}$.
\begin{theorem}
\label{THM:CVG_SED}
In the noiseless case, let ${\bm{h}}={\bm{z}}_t-{\bm{x}}$ and ${\bm{B}}_y={\bm{A}}_y+{\bm{A}}_y^T$. If ${\bm{z}}_t$ satisfies
\[
\|{\bm{h}}\|_2=\|{\bm{z}}_t-{\bm{x}}\|_2<\tau=\left(2-\frac{1}{q}\right)\cdot\sqrt{\frac{\lambda_{\bm{E}}}{4}}\,,
\]
where $q\in\big(\frac{1}{2},1\big)$ is some fixed constant and $\lambda_{\bm{E}}>0$ depends on the matrix ${\bm{E}}=\sum_{y=0}^{M-1}{\bm{B}}_y{\bm{x}}\vx^T{\bm{B}}_y^T$, the projected gradient descent update in \eqref{eq:pgd_update} converges linearly to ${\bm{x}}$,
\begin{equation}
\label{eq:converge}
\|{\bm{z}}_{t+1}-{\bm{x}}\|_2<\mu^{\frac{1}{2}}\|{\bm{z}}_t-{\bm{x}}\|_2\,,
\end{equation}
where $\mu\in(0,1)$ and the step size $\eta>0$ both depend on $\left\{q,\ \tau,\ {\bm{z}}_t\right\}$.
\end{theorem}
As proved in Appendix \ref{proof:thm:cvg_sed}, the size of the convergence neighbourhood varies for different signals ${\bm{x}}$. According to Lemma \ref{LEMMA:E_MIN}, $\lambda_{\bm{E}}$ can be computed via the following convex program:
\begin{align}
\begin{split}
\lambda_{\bm{E}} &= \min_{\widehat{{\bm{h}}}\in\mathcal{G}}\ \widehat{{\bm{h}}}^T{\bm{E}}\widehat{{\bm{h}}}=\min_{\widehat{{\bm{h}}}\in\mathcal{G}}\ \textstyle\sum_{y=0}^{M-1}\left(\widehat{{\bm{h}}}^T{\bm{B}}_y{\bm{x}}\right)^2\,,
\end{split}
\end{align}
where $\widehat{{\bm{h}}}=\frac{{\bm{z}}-{\bm{x}}}{\|{\bm{z}}-{\bm{x}}\|_1}$, ${\bm{z}}\in\mathcal{S}$, ${\bm{z}}\neq{\bm{x}}$, and $\mathcal{G}$ is the convex set defined in Lemma \ref{LEMMA:E_MIN}. Note that $\lambda_{\bm{E}}>0$ in the noiseless case. To see this, let us assume that $\lambda_{\bm{E}}=0$. We then have
\begin{align}
\label{eq:noiseless_lambda_E_zero}
({\bm{z}}-{\bm{x}})^T{\bm{B}}_y{\bm{x}}=0,\ \forall\ y\in\set{0,\cdots,M-1}.
\end{align}
Using ${\bm{B}}_0=2{\bm{I}}$, where ${\bm{I}}$ is the identity matrix, we can get ${\bm{z}}^T{\bm{x}}={\bm{x}}^T{\bm{x}}=N$. Since ${\bm{z}}\in\mathcal{S}$ and ${\bm{x}}$ is a binary vector containing exactly $N$ $1$-entries, the vector ${\bm{z}}$ must equal ${\bm{x}}$ to ensure ${\bm{z}}^T{\bm{x}}=N$. This is in contradiction with the assumption that ${\bm{z}}\neq{\bm{x}}$\footnote{If ${\bm{z}}={\bm{x}}$, then we already have a global optimal solution.}. Hence $\lambda_{\bm{E}}\neq 0$. Since ${\bm{E}}$ is a positive semidefinite matrix, we can get that $\lambda_{\bm{E}}>0$ and $\tau>0$.

\paragraph{Noisy recovery.} In this case we have ${\bm{x}}\in[0,1]^M$. From the proof of Theorem \ref{THM:CVG_SED}, we know that $\lambda_{\bm{E}}>0$ is a sufficient condition for the convergence neighbourhood to exist. We next show how the signal ${\bm{x}}$ plays a role in ensuring $\lambda_{\bm{E}}>0$. First, let us assume $\lambda_{\bm{E}}=0$. From \eqref{eq:noiseless_lambda_E_zero}, we have
\begin{align}
\label{eq:noisy_lambda_E_zero}
{\bm{x}}^T{\bm{B}}_y{\bm{z}} = {\bm{x}}^T{\bm{B}}_y{\bm{x}},\quad\forall\ y\in\set{0,\cdots,M-1}.
\end{align}
Let ${\bm{s}}_y^T={\bm{x}}^T{\bm{B}}_y$ and ${\bm{S}}^T=\left[{\bm{s}}_0\ {\bm{s}}_1\ \cdots\ {\bm{s}}_{M-1}\right]$, we can rewrite \eqref{eq:noisy_lambda_E_zero} as follows:
\begin{align}
\label{eq:noisy_R_z_x}
{\bm{S}}({\bm{z}}-{\bm{x}})={\bm{0}}\,.
\end{align}
Let $\mathcal{V}$ and $\mathcal{T}$ denote the following two sets
\begin{align}
\mathcal{V} &= \textnormal{Null}({\bm{S}})\ \backslash\ \set{{\bm{0}}}\\
\mathcal{T} &= \left\{{\bm{h}}\ |\ {\bm{h}}={\bm{z}}-{\bm{x}}\right\}\,.
\end{align}
If $\mathcal{V}\cap\mathcal{T}=\varnothing$, we can get that ${\bm{z}}={\bm{x}}$ according to \eqref{eq:noisy_R_z_x}. This is in contradiction with the assumption that ${\bm{z}}\neq{\bm{x}}$, hence $\lambda_{\bm{E}}\neq0$. Since ${\bm{E}}$ is positive semidefinite, we further have $\lambda_{\bm{E}}>0$ and $\tau>0$. We can see that $\mathcal{V}\cap\mathcal{T}=\varnothing$ is thus a sufficient condition for the convergence neighbourhood to exist.

\subsection{Noiseless Partial Digestion on Real Genome Data}

\begin{figure*}[t]
\centering
\includegraphics[height=2.5in]{figures/partial_digestion}
\caption{Partial digestion of a DNA with the restriction enzyme when $N=5$.}
\label{fig:partial_digestion}
\end{figure*}

As illustrated in Fig. \ref{fig:partial_digestion}, the DNA strands are partially digested at $N$ restriction sites by the restriction enzymes, producing all possible $\tbinom{N}{2}$ fragments. Here we perform experiments on \emph{E. Coli} K12 MG1655 genome data from the GenBank$^{\tiny{\text{\textregistered}}}$ assembly \cite{E:Coli:K12:MG1655}, which is a nucleotide sequence of length$=4,641,652$. Four letters \texttt{A}, \texttt{C}, \texttt{G}, \texttt{T} are used to represent the four nucleotide bases of the DNA strand \cite{NucleotidesSymbol:1985}. The list of restriction enzymes used in the experiments and the number of restriction sites (including the two ends \texttt{5'} and \texttt{3'} as dummy restriction sites) are shown in Table \ref{tab:restriction_enzyme}. Note that the recognition sequence could also be in a reverse order depending on which way the nucleotide sequence is read.

Since there are four nucleotide bases that cannot be further digested, the unlabeled pairwise distances are all integers in this case, and the DNA sequence has a total of $M=4,641,653$ equally spaced possible locations for the restriction sites. Note that the matrix ${\bm{A}}_y$ has a simple structure and thus needs not be stored during computation. Using our proposed approach, we can reconstruct all the locations of the sites in Table \ref{tab:restriction_enzyme} successfully.

\begin{table}[tbp]
\caption{The list of restriction enzymes used in the partial digest experiments.}
\label{tab:restriction_enzyme}
\centering
\begin{tabular}{llllll}
\toprule
Enzyme &Recognition sequence &$N$ &Enzyme &Recognition sequence &$N$ \\ \midrule
\multirow{2}{*}{SmaI} &\texttt{5'---CCC \textcolor{magenta}{|} GGG---3'} & \multirow{2}{*}{$495$} &\multirow{2}{*}{BamHI} &\texttt{5'---G \textcolor{magenta}{|} GATCC---3'} & \multirow{2}{*}{$512$}\\
&\texttt{3'---GGG \textcolor{magenta}{|} CCC---5'} & & &\texttt{3'---CCTAG \textcolor{magenta}{|} G---5'} &\\ \bottomrule
\end{tabular}
\vspace{-3mm}
\end{table}

\subsection{Turnpike Recovery on Simulated Data}
\label{sec:exp:turnpike}
In the turnpike recovery experiments where the points are located on a line, we compare the proposed approach and the state-of-the-art backtracking approach by \cite{Skiena:1994} through simulated noisy recovery experiments. We first uniformly sample $N=10$ points from the interval $[0,1]$ with the minimum pairwise distance between two different points set to $d_{\min}=1e^{-2}$ and the maximum pairwise distance set to $d_{\max}=1$. The length $L$ of the line $\boldsymbol l$ thus equals $d_{\max}$. The quantization step is set to $\Delta l=1e^{-3}$ to balance the trade-off between reducing the quantization error and computational complexity, creating $M=\frac{L}{\Delta l}=1e^3$ possible locations for the $10$ points. The distance measurement $d_k$ is corrupted with white Gaussian noise, $w\sim\mathcal{N}(0,\xi^2)$. We control the noise level by varying the standard deviation of the noise: $\xi\in\{0,1e^{-3}, 3e^{-3}, 5e^{-3}, 7e^{-3}, 9e^{-3}\}$. The results obtained when $\xi=0$ correspond to the case where there is only quantization error and no measurement noise.

For the proposed approach, the unlabeled pairwise distance measurements are collected and extended to form the multiset $\mathcal{D}$. As discussed in section \ref{subsec:ddm}, the parameter $\sigma$ in the approximated distribution $p(d)$ is unknown, and can be tuned in practice to obtain best performance. In the experiments, $\sigma$ is tuned in the interval $(0, d_{\min}=1e^{-2})$, producing multiple solutions corresponding to each $\sigma$. We shall choose the solution whose distance distribution is closest to the observed distance distribution in terms of the earth mover's distance \cite{EMD:2001}. The exact recovered point locations $\{\widehat{u}_1,\widehat{u}_2,\cdots,\widehat{u}_N\}$ are obtained using the aforementioned agglomerative clustering method in section \ref{subsec:ddm}. For each noise level specified by $\xi$, $100$ random simulations are performed and the number of correctly recovered points is recorded for each random run.

For the backtracking approach, the search path for every distance $d_k$ is performed in an interval $[d_k-\Delta d,\ d_k+\Delta d]$. In order to make a fair comparison, we need to ensure that both approaches are evaluating the distance $d_k$ within roughly the same range. Here we choose $\Delta d=5\sigma_{\max}=5e^{-2}$, where $\sigma_{\max}$ is the largest $\sigma$ tuned by the proposed approach. We should note that the best results are obtained by choosing $\Delta d=1$, i.e. the maximum pairwise distance. However, this essentially becomes performing an exhaustive search over all possible paths, and is simply impractical when the number of points $N$ and the number of possible locations $M$ are large. Since there are only $10$ points to be recovered in this case, we also compute the solution obtained via the exhaustive search as a comparison, which corresponds to the best solution one can hope to achieve given noisy measurements.

\begin{figure*}[tbp]
\centering
\subfigure{
\label{fig:n10_compare}
\includegraphics[height=2.2in]{figures/compare_methods_10.pdf}}
\subfigure{
\label{fig:n100_compare}
\includegraphics[height=2.2in]{figures/compare_methods_100.pdf}}
\caption{The distribution and the mean of the number of correctly recovered points across $100$ random runs in the \emph{``turnpike''} recovery experiments using the proposed approach ({\bfseries P}), the backtracking approach ({\bfseries B}) and the exhaustive search ({\bfseries E}). In each random run, $N$ points are uniformly sampled from the interval $[0,1]$. When $N=10$, the smallest distance between two different points is set to $d_{\min}=1e^{-2}$. When $N=100$, we set $d_{\min}=1e^{-4}$. The distances are further corrupted with white Gaussian noise $w\sim\mathcal{N}(0,\xi^2)$, where we control $\xi<d_{\min}$.}
\label{fig:tp_compare}
\end{figure*}

\begin{figure*}[tbp]
\centering
\subfigure{
\label{fig:n10_compare_bw}
\includegraphics[height=2.2in]{figures/compare_methods_10_bw.pdf}}
\subfigure{
\label{fig:n100_compare_bw}
\includegraphics[height=2.2in]{figures/compare_methods_100_bw.pdf}}
\caption{The distribution and the mean of the number of correctly recovered points across $100$ random runs in the \emph{``beltway''} recovery experiments using the proposed approach ({\bfseries P}). In each random run, $N$ points are uniformly sampled from a loop of length $L=d_{\min}+d_{\max}$, where the largest pairwise distance $d_{\max}$ is set to $1$, the smallest distance $d_{\min}$ between two different points is set to $1e^{-2}$ when $N=10$ and $1e^{-4}$ when $N=100$. The distances are further corrupted with white Gaussian noise $w\sim\mathcal{N}(0,\xi^2)$, where we control $\xi<d_{\min}$.}
\label{fig:bw_compare}
\end{figure*}

The recovered point locations can be matched to the true point locations efficiently using the Hungarian algorithm \cite{Hungarian:1955}. If the distance between a recovered location $\hat{u}_n$ and the true location $u_n$ is less than half the smallest pairwise distance $d_{\min}$, the recovery of the $n$-th point is considered to be a success. The recovery results when $N=10$ are shown in Fig. \ref{fig:tp_compare}: the distribution and the mean of the number of correctly recovered points across $100$ random runs are shown for each approach. We can see that the proposed approach is significantly more robust to noise compared to the conventional backtracking approach, and offers a competitive alternative to the  search approach.

In order to test how the two approaches are holding up against large-scale problems, we then uniformly sample $N=100$ points from the interval $[0,1]$ as before, with the minimum pairwise distance set to $d_{\min}=1e^{-4}$ and the maximum pairwise distance set to $d_{\max}=1$. The distance measurement $d_k$ is also corrupted with white Gaussian noise $w\sim\mathcal{N}(0,\xi^2)$, where $\xi\in\{0,1e^{-5}, 3e^{-5}, 5e^{-5}, 7e^{-5}, 9e^{-5}\}$. The quantization step is set to $\Delta l=1e^{-5}$, creating $M=\frac{L}{\Delta l}=1e^5$ possible locations for the $100$ points.

For the proposed approach, the standard deviation $\sigma$ in the noise model is tuned in the interval $\sigma\in(0,d_{\min}=1e^{-4})$. For the backtracking approach, the tolerance threshold $\tau_d$ is chosen to be $\tau_d=5\sigma_{\max}=5e^{-4}$. Since $N$ and $M$ are much larger in this case, we are not able to perform an exhaustive search for comparison here. The recovery results when $N=100$ are shown in Fig. \ref{fig:tp_compare}. We can see that the proposed approach is more robust and has greater advantage over the backtracking approach for large-scale problems. When the noise level is high, the backtracking approach is not able to produce solutions, whereas the proposed approach can still produce partially correct solutions.

\subsection{Beltway Recovery on Simulated Data}
\label{sec:exp:beltway}

We next use the proposed approach to perform the beltway recovery experiments where the points lie on a loop. To the best of our knowledge, our approach is the first practical approach that can solve the large-scale beltway problem efficiently. Note that the exhaustive search is impractical even when $N$ is small but $M$ is large \cite{Lemke2003}. Hence we only present the recovery results obtained using the proposed approach here. We uniformly sample $N$ points from a loop of length $L=d_{\min}+d_{\max}$, where $d_{\min}$ is the minimum distance between two different points and $d_{\max}$ is the maximum pairwise distance. When $N=10$, we set $d_{\min}=1e^{-2}$ and $d_{\max}=1$. The distance $d_k$ is also corrupted with a white Gaussian noise: $w_k\sim\mathcal{N}(0,\xi^2)$, where $\xi\in\set{0,1e^{-3}, 3e^{-3}, 5e^{-3},7e^{-3},9e^{-3}}$. The quantization step is set to $\Delta l=1e^{-3}$, creating $M=\frac{L}{\Delta l}=1.01e^3$ possible locations for the $10$-points case. When $N=100$, we set $d_{\min}=1e^{-4}$ and $d_{\max}=1$. The standard deviation of the white Gaussian noise is chosen from $\xi\in\set{0,1e^{-5},3e^{-5},5e^{-5},7e^{-5},9e^{-5}}$ as before, and the quantization step is set to $\Delta l=1e^{-5}$, creating $M=\frac{L}{\Delta l}=1.0001e^5$ possible locations for the $100$-points case. The recovery results are shown in Fig. \ref{fig:bw_compare}. We can see that the proposed approach is able to reconstruct all the point locations correctly when there is only quantization error and no measurement noise, i.e. $\xi=0$. When measurement noise is added to the distance, the proposed approach could still perform a partial reconstruction.

\subsection{Comparison of Initialization Schemes}

\begin{figure*}[tbp]
\centering
\subfigure{
\label{fig:init_tp_n100_compare}
\includegraphics[width=.48\textwidth]{figures/compare_methods_init_100.pdf}}
\subfigure{
\label{fig:init_bw_n100_compare}
\includegraphics[width=.48\textwidth]{figures/compare_methods_init_100_bw.pdf}}
\caption{The distribution and the mean of the number of correctly recovered points across $100$ random runs in the turnpike and beltway recoveries comparing the \emph{``three initialization schemes''}: the spectral initialization ({\bfseries S}), the random initialization ({\bfseries R}), and the uniform initialization ({\bfseries U}). In each random run, $N=100$ points are uniformly sampled from the interval $[0,1]$, with the smallest distance between two different points set to $d_{\min}=1e^{-4}$. The distances are further corrupted with white Gaussian noise $w\sim\mathcal{N}(0,\xi^2)$, where we control $\xi<d_{\min}$.}
\label{fig:init_compare}
\end{figure*}

A spectral initialization scheme is adopted in the proposed approach to solve the nonconvex turnpike and beltway recoveries. It is meant to provide a good initializer that highlights the possible point locations. Here we put it to test and compare it with the other two initialization schemes, i.e. the ``random'' initialization and the ``uniform'' initialization. In the random initialization scheme, the entries of the initializer ${\bm{z}}_0$ are generated independently according to the white Gaussian distribution $\mathcal{N}(0,0.01)$. In the uniform initialization scheme, the entries of ${\bm{z}}_0$ are set to all ones. We should note that the initializers from all three schemes are projected to the convex set $\mathcal{S}$ defined by \eqref{eq:relaxed_constraint} and \eqref{eq:l1_constraint} before they can be used with the projected gradient descent. Following the same settings when $N=100$ as in sections \ref{sec:exp:turnpike} and \ref{sec:exp:beltway}, we perform simulated noisy turnpike and beltway recoveries using the three initialization schemes. The recovery results are shown in Fig. \ref{fig:init_compare}. For the turnpike recovery, the spectral initialization is more robust than the other two schemes. For the beltway recovery, the spectral initialization and the random initialization perform almost equally well, and they both perform better than the uniform initialization.

\section{Proofs for Projected Gradient Descent}

\remove{
\subsection{Proof of Lemma ~\ref{LEMMA:VALID_STEPSIZE}}
\label{proof:lemma:valid_stepsize}

Since ${\bm{z}}_t\in\mathcal{S}$, we can represent any point ${\bm{v}}$ in the convex set $\mathcal{S}$ by
\begin{align}
{\bm{v}}&={\bm{z}}_t+{\bm{r}}\,,
\end{align}
where $\sum_{m=1}^M r_m=0$, $-{z_t}_m\leq r_m\leq 1-{z_t}_m$, $\forall\ m\in\set{1,\cdots,M}$ and ${z_t}_m$ is the $m$-th entry of ${\bm{z}}_t$.
\begin{enumerate}[label={\arabic*)}]
\item To simplify notation, we use $f^\prime$ to denote the gradient $\nabla f({\bm{z}}_t)$. The projection of ${\bm{z}}_t-\eta\cdot f^\prime$ onto $\mathcal{S}$ can be obtained via the following two equivalent problems:
\begin{align}
\begin{split}
\arg\min_{{\bm{v}}\in\mathcal{S}}\quad\|{\bm{v}}-({\bm{z}}_t-\eta\cdot f^\prime)\|_2^2 \quad \Longleftrightarrow \quad \arg\min_{{\bm{r}}}\quad\|{\bm{r}}+\eta\cdot f^\prime\|_2^2
\end{split}
\end{align}
The gradient $f^\prime$ can be decomposed into the sum of two vectors $f^\prime_1$ and $f^\prime_2$ as follows. Let $\boldsymbol 1$ denote a vector of length $M$ with all $1$s. We have
\begin{align}
f^\prime_1&=\frac{{\boldsymbol 1}^Tf^\prime}{M}\cdot\boldsymbol 1\\
f^\prime_2&=f^\prime-f^\prime_1\,.
\end{align}
Since ${\bm{r}}^Tf^\prime_1=0$, we get:
\begin{align}
\begin{split}
\widehat{{\bm{r}}}=&\arg\min_{{\bm{r}}}\quad\|{\bm{r}}+\eta\cdot(f^\prime_1+f^\prime_2)\|_2^2\\
=&\arg\min_{{\bm{r}}}\quad{\bm{r}}^T{\bm{r}}+2\eta\cdot{\bm{r}}^Tf^\prime_2\,.
\end{split}
\end{align}

Since we can always make ${\bm{r}}^T{\bm{r}}+2\eta\cdot{\bm{r}}^Tf^\prime_2=0$ by setting ${\bm{r}}=\boldsymbol 0$, we then have the following inequality for the minimizing $\widehat{{\bm{r}}}$:
\begin{align}
\label{eq:convergence_ieq1}
\widehat{{\bm{r}}}^T\widehat{{\bm{r}}}+2\eta\cdot\widehat{{\bm{r}}}^Tf^\prime_2\leq 0\,.
\end{align}
Since $\widehat{{\bm{r}}}^T\widehat{{\bm{r}}}\geq 0$ and $\eta>0$, we must have:
\begin{align}
\label{eq:ineq_1}
\widehat{{\bm{r}}}^Tf^\prime_2\leq 0\,.
\end{align}

\item Let ${\bm{B}}_y={\bm{A}}_y+{\bm{A}}_y^T$. We can compute $\nabla^2 f({\bm{z}})$ as follows
\begin{align}
\nabla^2 f({\bm{z}}) = \frac{2}{MK^2}\sum_{y=0}^{M-1}\left[{\bm{B}}_y{\bm{z}}\vz^T{\bm{B}}_y^T+\left({\bm{z}}^T{\bm{A}}_y{\bm{z}}-{\bm{x}}^T{\bm{A}}_y{\bm{x}}\right)\cdot{\bm{B}}_y\right]\,.
\end{align}

Since $\nabla^2 f({\bm{z}}_t)$ is bounded, $\nabla f({\bm{z}}_t)$ is locally Lipschitz continuous in a neighbourhood $\rho({\bm{z}}_t)$ around ${\bm{z}}_t$. In other words, $\forall {\bm{v}}\in\rho({\bm{z}}_t)$, the objective function $f({\bm{v}})$ has Lipschitz continuous gradient. The projection of ${\bm{z}}_t$ onto $\mathcal{S}$ can be computed as ${\bm{s}}={\bm{z}}_t+\widehat{{\bm{r}}}$. We can then obtain a quadratic upper bound on $f({\bm{s}})$. Suppose the Lipschitz constant is $\mathcal{L}$. We have
\begin{align}
\label{eq:ub_fvh}
\begin{split}
f({\bm{s}})&=f\left({\bm{z}}_t+\widehat{{\bm{r}}}\right)\\
&\leq f\left({\bm{z}}_t\right)+\widehat{{\bm{r}}}^Tf^\prime+\frac{\mathcal{L}}{2}\widehat{{\bm{r}}}^T\widehat{{\bm{r}}}\\
&=f\left({\bm{z}}_t\right)+\widehat{{\bm{r}}}^T(f^\prime_1+f^\prime_2)+\frac{\mathcal{L}}{2}\widehat{{\bm{r}}}^T\widehat{{\bm{r}}}\\
&=f\left({\bm{z}}_t\right)+\widehat{{\bm{r}}}^Tf^\prime_2+\frac{\mathcal{L}}{2}\widehat{{\bm{r}}}^T\widehat{{\bm{r}}}\,.
\end{split}
\end{align}

If the step size $\eta$ is chosen such that $0<\eta\leq\frac{1}{\mathcal{L}}$, using \eqref{eq:ineq_1}, we have:
\begin{align}
\label{eq:ll_2_5}
\frac{2}{\mathcal{L}}\cdot\widehat{{\bm{r}}}^Tf^\prime_2 \leq 2\eta\cdot\widehat{{\bm{r}}}^Tf^\prime_2 \,.
\end{align}
Plug \eqref{eq:ll_2_5} into (\ref{eq:convergence_ieq1}). We can get:
\begin{align}
\label{eq:ub_fvh_p2}
\widehat{{\bm{r}}}^Tf^\prime_2+\frac{\mathcal{L}}{2}\widehat{{\bm{r}}}^T\widehat{{\bm{r}}}\leq 0\,.
\end{align}
Combining \eqref{eq:ub_fvh} and \eqref{eq:ub_fvh_p2}, we have $f({\bm{s}})\leq f({\bm{z}}_t)$.
\end{enumerate}
}

\subsection{Proof of Lemma \ref{lemma:upper_bound}}
\label{proof:lemma:upper_bound}
Suppose that $s_i<1$. We construct a vector $\widetilde{{\bm{s}}}\in\mathbb{R}^M$ out of ${\bm{s}}$ by swapping the positions of $s_i$ and $s_j$ in ${\bm{s}}$, i.e. $\widetilde{s}_i=s_j$ and $\widetilde{s}_j=s_i$. $\widetilde{{\bm{s}}}$ also satisfies the constraints in (\ref{eq:projection_box_constrains}). Since $z_i>z_j$ and $s_j=1$, we then have:
\begin{align}
\begin{split}
\|{\bm{s}}-{\bm{z}}\|_2^2-\|\tilde{{\bm{s}}}-{\bm{z}}\|_2^2&=(s_i-z_i)^2+(s_j-z_j)^2-(s_j-z_i)^2-(s_i-z_j)^2\\
&=2(1-s_i)(z_i-z_j)\\
&> 0\,.
\end{split}
\end{align}
This is in contradiction with the fact that ${\bm{s}}$ is the minimizer of (\ref{eq:projection_box_constrains}). Hence $s_i$ must be $1$.

\subsection{Proof of Lemma \ref{LEMMA:UNIQUE_R}}
\label{proof:lemma:unique_r}
Let $\mathcal{S}$ denote the convex set defined by the constraints $0\leq s_m\leq 1$, $\forall\ 1\leq m\leq M$ and $\|{\bm{s}}\|_1=N$. Note that the entries of ${\bm{z}}$ and ${\bm{x}}$ are in a non-increasing order. We will proceed in the following two steps:
\begin{enumerate}[label={\arabic*)}]
\item Since $\mathcal{S}$ is non-empty, the projection onto it exists, i.e. there is one $r\in\set{1,\cdots,N}$ that produces the $(\rho,\kappa)$ that satisfy $1\leq z_{r-1}-\kappa$ if $2\leq r\leq N<\rho$ and $0<z_r-\kappa<1$. In fact, since $\mathcal{S}$ is a closed convex set, the projection is also unique.
\item Without loss of generality, suppose that there are two different $r_1<r_2\in\set{1,\cdots,N}$ that produce the two pairs $(\rho_1,\kappa_1|r_1)$ and $(\rho_2,\kappa_2|r_2)$ that satisfy the constraints (\ref{eq:cst_r}) and (\ref{eq:cst_rm1}). We have:
\begin{gather*}
r_1<r_2 \,\Rightarrow\, r_1\leq r_2-1 \,\Rightarrow\, z_{r_2-1}-\kappa_1\leq z_{r_1}-\kappa_1< 1 \,\Rightarrow\, z_{r_2-1}-1< \kappa_1\\
1< z_{r_2-1}-\kappa_2 \,\Rightarrow\, \kappa_2< z_{r_2-1}-1\,.
\end{gather*}
Hence $\kappa_2<\kappa_1$. We further have:
\begin{gather*}
z_{\rho_1}-\kappa_1>0 \,\Rightarrow\, z_{\rho_1}>\kappa_1\\
z_{\rho_2+1}-\kappa_2\leq 0 \,\Rightarrow\, z_{\rho_2+1}\leq\kappa_2\\
z_{\rho_2+1}\leq\kappa_2<\kappa_1< z_{\rho_1} \,\Rightarrow\, z_{\rho_2+1} < z_{\rho_1}\,.
\end{gather*}
Hence $\rho_2+1>\rho_1\,\Rightarrow\,\rho_2\geq\rho_1$.
\begin{enumerate}
\item If $r_2\leq\rho_1$, we can find the upper bound for the sum of the first $\rho_1$ entries of ${\bm{s}}_1$:
\begin{align}
\label{eq:r_rho_neq_1}
\begin{split}
r_1-1+\sum_{m=r_1}^{\rho_1}(z_m-\kappa_1)=\,&r_1-1+\sum_{m=r_1}^{r_2-1}(z_m-\kappa_1)+\sum_{m=r_2}^{\rho_1}(z_m-\kappa_1)\\
<\,&r_1-1+\sum_{m=r_1}^{r_2-1}1+\sum_{m=r_2}^{\rho_1}(z_m-\kappa_1)\\
<\,&r_2-1+\sum_{m=r_2}^{\rho_1}(z_m-\kappa_2)\\
\leq\,&r_2-1+\sum_{m=r_2}^{\rho_2}(z_m-\kappa_2)\,.
\end{split}
\end{align}

\item If $r_2>\rho_1$, we can compute:
\begin{align}
\label{eq:r_rho_neq_2}
\begin{split}
r_1-1+\sum_{m=r_1}^{\rho_1}(z_m-\kappa_1)\,\leq\, &r_1-1+\sum_{m=r_1}^{\rho_1}1\\
=\,&\rho_1\\
\leq\,&r_2-1\\
<\,&r_2-1+\sum_{m=r_2}^{\rho_2}(z_m-\kappa_2)\,.
\end{split}
\end{align}
\end{enumerate}

Let ${\bm{s}}_1,{\bm{s}}_2$ denote the solutions of \eqref{eq:projection_box_constrains} produced by $r_1,r_2$ respectively. Both (\ref{eq:r_rho_neq_1}) and (\ref{eq:r_rho_neq_2}) show that $\|{\bm{s}}_1\|_1<\|{\bm{s}}_2\|_1$. This is in contradiction with the assumption that they have the same $l_1$ norm, i.e. $\|{\bm{s}}_1\|_1=\|{\bm{s}}_2\|_1=N$. Hence $r_1=r_2$, there is only one $r\in\set{1,\cdots,N}$ that produces the $(\rho,\kappa)$ that satisfy the constraints (\ref{eq:cst_r}) and (\ref{eq:cst_rm1}).
\end{enumerate}

\section{Proofs for Convergence Analysis}

\subsection{Lemma ~\ref{LEMMA:E_MIN}}
\label{proof:lemma:e_min}

\begin{lemma}
\label{LEMMA:E_MIN}
Let ${\bm{B}}_y={\bm{A}}_y+{\bm{A}}_y^T$, ${\bm{E}}=\sum_{y=0}^{M-1}{\bm{B}}_y{\bm{x}}{{\bm{x}}}^T{\bm{B}}_y^T$, $\mathcal{S}$ be the convex set defined by the constraints \eqref{eq:relaxed_constraint},\eqref{eq:l1_constraint}. The following problem is convex $\forall\ {\bm{z}}\in\mathcal{S}$, ${\bm{z}}\neq{\bm{x}}$:
\begin{align}
\label{eq:lambda_min_val}
\begin{split}
\lambda({\bm{E}}) &= \min_{{\bm{z}}\in\mathcal{S}, {\bm{z}}\neq{\bm{x}}}\,\frac{1}{\|{\bm{z}}-{\bm{x}}\|_1^2}({\bm{z}}-{\bm{x}})^T{\bm{E}}({\bm{z}}-{\bm{x}})\\
&=\min_{\widehat{{\bm{h}}}\in\mathcal{G}}\,\widehat{{\bm{h}}}^T{\bm{E}}\widehat{{\bm{h}}}\,,
\end{split}
\end{align}
where $\lambda({\bm{E}})>0$ and $\widehat{{\bm{h}}}=\frac{{\bm{z}}-{\bm{x}}}{\|{\bm{z}}-{\bm{x}}\|_1}$, $\mathcal{G}$ is a convex set defined by the following constraints:
\begin{align}
\label{eq:hh_con_2}
&\sum_{i=1}^M\widehat{h}_i=0\\
\label{eq:hh_con_3}
&\widehat{h}_i\in[0,\,0.5]\quad\textnormal{if $x_i=0$}\\
\label{eq:hh_con_4}
&\widehat{h}_i\in[-0.5,\,0]\quad\textnormal{if $x_i=1$}\\
\label{eq:hh_con_1}
&\|\widehat{{\bm{h}}}\|_1={\bm{r}}^T\widehat{{\bm{h}}}=1\,,
\end{align}
where ${\bm{r}}\in\{-1,1\}^M$ depends on ${\bm{x}}$ and is defined as follows:
\begin{align}
r_i=\left\{
\begin{array}{l}
1\\
-1
\end{array}
\quad
\begin{array}{l}
\textnormal{if $x_i=0$}\\
\textnormal{if $x_i=1$}\,.
\end{array}
\right.
\end{align}
\end{lemma}

\begin{proof}
Since ${\bm{E}}=\sum_{y=0}^{M-1}{\bm{B}}_y{\bm{x}}{{\bm{x}}}^\textrm{T}{\bm{B}}_y^\textrm{T}$, we can see that $\widehat{{\bm{h}}}^T{\bm{E}}\widehat{{\bm{h}}}=\sum_y\left(\widehat{{\bm{h}}}^T{\bm{B}}_y{\bm{x}}\right)^2\geq 0$, $\forall\ \widehat{{\bm{h}}}\in\mathbb{R}^M$. Hence ${\bm{E}}$ is positive-semidefinite. We define the following set $\mathcal{H}$:
\begin{align}
\label{eq:H_set_def}
\mathcal{H}=\left\{\widehat{{\bm{h}}}\,\left|\,\widehat{{\bm{h}}}=\frac{1}{\|{\bm{z}}-{\bm{x}}\|_1}({\bm{z}}-{\bm{x}}),\quad\forall {\bm{z}}\in\mathcal{S}\,,{\bm{z}}\neq{\bm{x}}\right.\right\}\,.
\end{align}
where $\mathcal{S}$ is the convex set defined by \eqref{eq:relaxed_constraint} and \eqref{eq:l1_constraint}. We then have
\begin{align}
\begin{split}
    \lambda({\bm{E}}) &= \min_{{\bm{z}}\in\mathcal{S}, {\bm{z}}\neq{\bm{x}}}\,\frac{1}{\|{\bm{z}}-{\bm{x}}\|_1^2}({\bm{z}}-{\bm{x}})^T{\bm{E}}({\bm{z}}-{\bm{x}})\\
    & = \min_{\widehat{h}\in\mathcal{H}}\,\widehat{{\bm{h}}}^T{\bm{E}}\widehat{{\bm{h}}}\,.
\end{split}
\end{align}

\begin{enumerate}[label={\arabic*)}]
\item We first prove that $\mathcal{H}$ is a convex set. Let $\widehat{{\bm{h}}}^{(1)},\widehat{{\bm{h}}}^{(2)}\in\mathcal{H}$. We have $\textnormal{sign}(\widehat{h}_i^{(1)})=\textnormal{sign}(\widehat{h}_i^{(2)})$ if $\widehat{h}_i^{(1)}\neq0,\ \widehat{h}_i^{(2)}\neq0$. Let $\widehat{{\bm{h}}}^{(3)}=(1-\rho)\widehat{{\bm{h}}}^{(1)}+\rho\widehat{{\bm{h}}}^{(2)}$, where $\rho\in(0,1)$. We have:
\begin{align}
\label{eq:l1_norm_z3}
\begin{split}
\|\widehat{{\bm{h}}}^{(3)}\|_1 & = \left\|(1-\rho)\widehat{{\bm{h}}}^{(1)}+\rho\widehat{{\bm{h}}}^{(2)}\right\|_1\\
&=\sum_{i=1}^M\left|(1-\rho)\widehat{h}_i^{(1)}+\rho \widehat{h}_i^{(2)}\right|\\
&=\sum_{i=1}^M\left|(1-\rho)\widehat{h}_i^{(1)}\right|+\left|\rho \widehat{h}_i^{(2)}\right|\\
&=(1-\rho)\left\|\widehat{{\bm{h}}}^{(1)}\right\|_1+\rho\left\|\widehat{{\bm{h}}}^{(2)}\right\|_1=1\,.
\end{split}
\end{align}
Let $\nu_1=\frac{1-\rho}{\left\|{\bm{z}}^{(1)}-{\bm{x}}\right\|_1}$, $\nu_2=\frac{\rho}{\left\|{\bm{z}}^{(2)}-{\bm{x}}\right\|_1}$. We have
\begin{align}
\begin{split}
\widehat{{\bm{h}}}^{(3)}&=(1-\rho)\widehat{{\bm{h}}}^{(1)}+\rho \widehat{{\bm{h}}}^{(2)}\\
&=\frac{1-\rho}{\left\|{\bm{z}}^{(1)}-{\bm{x}}\right\|_1}\left({\bm{z}}^{(1)}-{\bm{x}}\right)+\frac{\rho}{\left\|{\bm{z}}^{(2)}-{\bm{x}}\right\|_1}\left({\bm{z}}^{(2)}-{\bm{x}}\right)\\
&=\left(\nu_1+\nu_2\right)\left(\frac{\nu_1}{\nu_1+\nu_2}{\bm{z}}^{(1)}+\frac{\nu_2}{\nu_1+\nu_2}{\bm{z}}^{(2)}-{\bm{x}}\right)\\
&=(\nu_1+\nu_2)({\bm{z}}^{(3)}-{\bm{x}})\,.
\end{split}
\end{align}
Using \eqref{eq:l1_norm_z3}, we can see that $\nu_1+\nu_2=\frac{1}{\|{\bm{z}}^{(3)}-{\bm{x}}\|_1}$. Since ${\bm{z}}^{(1)},{\bm{z}}^{(2)}\in\mathcal{S}$, we have ${\bm{z}}^{(3)}\in\mathcal{S}$. We have shown that $\widehat{{\bm{h}}}^{(3)}$ can be written in the same form given in \eqref{eq:H_set_def} and thus belongs to $\mathcal{H}$.
\begin{align}
    \widehat{{\bm{h}}}^{(3)}=\frac{1}{\|{\bm{z}}^{(3)}-{\bm{x}}\|_1}({\bm{z}}^{(3)}-{\bm{x}})\,.
\end{align}
Hence $\widehat{{\bm{h}}}^{(3)}\in\mathcal{H}$, and $\mathcal{H}\subset\mathbb{R}^M$ is a convex set. Minimizing $\widehat{{\bm{h}}}^\textrm{T}{\bm{E}}\widehat{{\bm{h}}}$ with respect to $\widehat{{\bm{h}}}\in\mathcal{H}$ is a convex problem.

\item We next prove that $\lambda({\bm{E}})$ in \eqref{eq:lambda_min_val} is strictly positive. If $\widehat{{\bm{h}}}^T{\bm{E}}\widehat{{\bm{h}}}=\sum_y\left(\widehat{{\bm{h}}}^T{\bm{B}}_y{\bm{x}}\right)^2=0$, we have $({\bm{z}}-{\bm{x}})^T{\bm{B}}_y{\bm{x}}=0$, $\forall\ y\in\set{0,\cdots,M-1}$. When $y=0$, ${\bm{B}}_0=2{\bm{I}}$, where ${\bm{I}}$ is the identity matrix, we get ${\bm{z}}^T{\bm{x}}={{\bm{x}}}^\textrm{T}{\bm{x}}=N$. Since ${\bm{z}}\in\mathcal{S}$ and ${\bm{x}}\in\set{0,1}^M$ in the noiseless case, we have ${\bm{z}}={\bm{x}}$. This is in contradiction with the assumption ${\bm{z}}\neq{\bm{x}}$, hence $\widehat{{\bm{h}}}^T{\bm{E}}\widehat{{\bm{h}}}>0$, $\forall\ \widehat{{\bm{h}}}\in\mathcal{H}$.

\item We finally prove that $\mathcal{H}$ and a new set $\mathcal{G}$ defined by the following constraints are the same:
\begin{align}
\label{eq:hh_con_2}
&\sum_{i=1}^M\widehat{h}_i=0\\
\label{eq:hh_con_3}
&\widehat{h}_i\in[0,\,0.5]\quad\textnormal{if $x_i=0$}\\
\label{eq:hh_con_4}
&\widehat{h}_i\in[-0.5,\,0]\quad\textnormal{if $x_i=1$}\\
\label{eq:hh_con_1}
&\|\widehat{{\bm{h}}}\|_1={\bm{r}}^T\widehat{{\bm{h}}}=1\,,
\end{align}
where ${\bm{r}}\in\{-1,1\}^M$ is defined as follows:
\begin{align}
r_i=\left\{
\begin{array}{l}
1\\
-1
\end{array}
\quad
\begin{array}{l}
\textnormal{if $x_i=0$}\\
\textnormal{if $x_i=1$}\,.
\end{array}
\right.
\end{align}
\begin{itemize}
\item It is easy to verify that if $\widehat{{\bm{h}}}\in\mathcal{H}$, \eqref{eq:hh_con_2} and \eqref{eq:hh_con_1} hold. Since $z_i\in[0,1]$ and $x_i\in\{0,1\}$, if $x_i=0$, $\widehat{h}_i\geq 0$; if $x_i=1$, $\widehat{h}_i\leq 0$. On the other hand, if $|\widehat{h}_i|>0.5$, from \eqref{eq:hh_con_2} we have $\sum_{j\neq i}|\widehat{h}_j|\geq|\sum_{j\neq i}\widehat{h}_j|=|-\widehat{h}_i|>0.5$. This means that $\|\widehat{{\bm{h}}}\|_1=|\widehat{h}_i|+\sum_{j\neq i}|\widehat{h}_j|>1$, which contradicts \eqref{eq:hh_con_1}. Hence $|\widehat{h}_i|\leq 0.5$, \eqref{eq:hh_con_3} and \eqref{eq:hh_con_4} hold. This proves that $\widehat{{\bm{h}}}\in\mathcal{G}$.
\item If $\widehat{{\bm{h}}}\in\mathcal{G}$, we can construct such a $\widehat{{\bm{z}}}={\bm{x}}+\widehat{{\bm{h}}}$. It is easy to verify that $\widehat{{\bm{z}}}\in\mathcal{S}$ and $\|\widehat{{\bm{z}}}-{\bm{x}}\|_1=\|\widehat{{\bm{h}}}\|_1=1$. Hence $\widehat{{\bm{h}}} = \frac{1}{\|\widehat{{\bm{z}}}-{\bm{x}}\|_1}(\widehat{{\bm{z}}}-{\bm{x}})\in\mathcal{H}$.
\end{itemize}
\end{enumerate}
Computing $\lambda({\bm{E}})=\min_{\widehat{{\bm{h}}}\in\mathcal{G}}\widehat{{\bm{h}}}^T{\bm{E}}\widehat{{\bm{h}}}\,>0$ is thus a convex problem, and can be efficiently solved via quadratic programming.
\end{proof}

\subsection{Proof of Theorem ~\ref{THM:CVG_SED}}
\label{proof:thm:cvg_sed}
Let ${\bm{B}}_y={\bm{A}}_y+{\bm{A}}_y^T$. The objective function $f({\bm{z}})$ in \eqref{eq:constrained_nonconvex} can be written as
\begin{align}
    f({\bm{z}}) = \frac{1}{4MK^2}\sum_{y=0}^{M-1}\left({\bm{z}}^T{\bm{B}}_y{\bm{z}}-{\bm{x}}^T{\bm{B}}_y{\bm{x}}\right)^2\,.
\end{align}
The gradient $\nabla f({\bm{z}})$ is
\begin{align}
\begin{split}
    \nabla f({\bm{z}}) &= \frac{1}{MK^2}\sum_{y=0}^{M-1}{\bm{B}}_y{\bm{z}}\cdot\left({\bm{z}}^T{\bm{B}}_y{\bm{z}}-{\bm{x}}^T{\bm{B}}_y{\bm{x}}\right)\\
    &= \frac{1}{MK^2}\sum_{y=0}^{M-1}{\bm{B}}_y{\bm{z}}\cdot({\bm{z}}-{\bm{x}})^T{\bm{B}}_y({\bm{z}}+{\bm{x}})\,.
\end{split}
\end{align}
In order to prove the conditions under which the projected gradient descent updates converge to a global optimum, we first establish the conditions on the gradient descent updates.

\paragraph{Step 1:} When the distance between the current solution ${\bm{z}}_t$ and a global optimum ${\bm{x}}$ is less than some $\tau>0$, i.e. $\|{\bm{z}}_t-{\bm{x}}\|_2<\tau$, we would like to show that the gradient descent update ${\bm{z}}_t-\eta\nabla f({\bm{z}}_t)$ converges linearly to ${\bm{x}}$. In other words, we need to prove the following \eqref{eq:diff} is less than $0$ for some $\mu\in(0,1)$ and $\eta>0$:
\begin{align}
\label{eq:diff}
\begin{split}
\left\|{\bm{z}}_t-\eta\nabla f({\bm{z}}_t)-{\bm{x}} \right\|^2_2-\mu\|{\bm{z}}_t-{\bm{x}}\|^2_2
=\eta^2\|\nabla f({\bm{z}}_t)\|_2^2-2\eta\langle{\bm{z}}_t-{\bm{x}},\, \nabla f({\bm{z}}_t)\rangle+(1-\mu)\|{\bm{z}}_t-{\bm{x}}\|_2^2\,.
\end{split}
\end{align}
We then have:
\begin{align}
\label{eq:first_term_lb}
\begin{split}
\|\nabla f({\bm{z}}_t) \|_2^2& = \frac{1}{K^4}\left\|\frac{1}{M}\sum_y{\bm{B}}_y{\bm{z}}_t\cdot({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\right\|_2^2\\
&\leq \frac{1}{MK^4}\sum_y\left\|{\bm{B}}_y{\bm{z}}_t\cdot({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\right\|_2^2\\
&= \frac{1}{MK^4}\sum_y\left\|{\bm{B}}_y{\bm{z}}_t\right\|_2^2\cdot\left(({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\right)^2\\
&\leq \frac{1}{MK^4}\sum_y\sigma_{\max}^2\left({\bm{B}}_y\right)\|{\bm{z}}_t\|_2^2\cdot\left(({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\right)^2\\
&\leq \frac{4}{MK^4}\|{\bm{z}}_t\|_2^2\sum_y\left(({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\right)^2\\
&= \frac{16}{K^2}\|{\bm{z}}_t\|_2^2\cdot f({\bm{z}}_t)\,,
\end{split}
\end{align}
where $\sigma_{\max}^2\left({\bm{B}}_y\right)\leq 4$, $\forall\ y=\{0,1,\cdots,M-1\}$ according to the Schur's bound \cite{Schur1911}.

Using the Cauchy-Schwarz inequality, we also have that
\begin{align}
\label{eq:second_term}
\begin{split}
\langle{\bm{z}}_t-{\bm{x}},\, \nabla f({\bm{z}}_t)\rangle&=\frac{1}{MK^2}\sum_y({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y{\bm{z}}_t\cdot({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\\
&=4f({\bm{z}}_t)-\frac{1}{MK^2}\sum_y({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\cdot({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y{\bm{x}}\\
&\geq 4f({\bm{z}}_t)-\sqrt{4f({\bm{z}}_t)}\sqrt{\frac{1}{MK^2}\sum_y\left(({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y{\bm{x}}\right)^2}\,.
\end{split}
\end{align}
We proceed by further lower-bounding the above \eqref{eq:second_term}. Let ${\bm{h}}={\bm{z}}_t-{\bm{x}}$. For some $\frac{1}{2}<q<1$, we have
\begin{align}
\label{eq:second_term_lb1}
\begin{split}
&q^2\sum_y\left(({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y({\bm{z}}_t+{\bm{x}})\right)^2-\sum_y\left(({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y{\bm{x}}\right)^2\\
=\ &q^2\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y({\bm{h}}+2{\bm{x}})\right)^2-\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{x}}\right)^2\\
=\ &q^2\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2+4q^2\sum_y{\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\cdot{\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{x}}+(4q^2-1)\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{x}}\right)^2\\
\geq \ &q^2\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2-4q^2\sqrt{\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2}\sqrt{\sum_y\left({\bm{h}}^T{\bm{B}}_y{\bm{x}}\right)^2}+(4q^2-1)\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{x}}\right)^2\\
=\ &\left(q\sqrt{\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2}-2q\sqrt{\sum_y\left({\bm{h}}^T{\bm{B}}_y{\bm{x}}\right)^2}\right)^2-\left(\sqrt{\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{x}}\right)^2}\right)^2\\
=\ &\left(q\sqrt{\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2}-(2q-1)\sqrt{\sum_y\left({\bm{h}}^T{\bm{B}}_y{\bm{x}}\right)^2}\right)\left(q\sqrt{\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2}-(2q+1)\sqrt{\sum_y\left({\bm{h}}^T{\bm{B}}_y{\bm{x}}\right)^2}\right)\,.
\end{split}
\end{align}
To make \eqref{eq:second_term_lb1} greater than $0$, either of the following two inequalities should hold:
\begin{align}
\label{eq:second_term_lb1_one}
\sqrt{\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2}&>(2+\frac{1}{q})\sqrt{\sum_y\left({\bm{h}}^T{\bm{B}}_y{\bm{x}}\right)^2}\\
\label{eq:second_term_lb1_two}
\sqrt{\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2}&<(2-\frac{1}{q})\sqrt{\sum_y\left({\bm{h}}^T{\bm{B}}_y{\bm{x}}\right)^2}\,.
\end{align}
We can obtain an upper bound on $\|{\bm{h}}\|_2$ to make \eqref{eq:second_term_lb1_two} hold. Specifically, the left-hand side of \eqref{eq:second_term_lb1_two} can be upper bounded via:
\begin{align}
\label{eq:ub_hDh}
\begin{split}
\sum_y\left({\bm{h}}^\textrm{T}{\bm{B}}_y{\bm{h}}\right)^2&=\|{\bm{h}}\|_2^4\cdot\sum_y\left({\bm{u}}^\textrm{T}{\bm{B}}_y{\bm{u}}\right)^2\\
&=\|{\bm{h}}\|_2^4\cdot\sum_y\|{\bm{B}}_y\|_{op}^2\cdot\left(\frac{|{\bm{u}}^T{\bm{B}}_y{\bm{u}}|}{\|{\bm{B}}_y\|_{op}}\right)^2\\
&\leq \|{\bm{h}}\|_2^4\cdot\sum_y\|{\bm{B}}_y\|_{op}^2\cdot\left(\frac{|{\bm{u}}^T{\bm{B}}_y{\bm{u}}|}{\|{\bm{B}}_y\|_{op}}\right)\\
&=\|{\bm{h}}\|_2^4\cdot\sum_y\|{\bm{B}}_y\|_{op}\cdot|{\bm{u}}^T{\bm{B}}_y{\bm{u}}|\\
&\leq \|{\bm{h}}\|_2^4\cdot\sum_y\|{\bm{B}}_y\|_{op}\cdot |{\bm{u}}|^T{\bm{B}}_y|{\bm{u}}|\\
&=\|{\bm{h}}\|_2^4\cdot\sum_y\sigma_{\max}({\bm{B}}_y)\cdot |{\bm{u}}|^T{\bm{B}}_y|{\bm{u}}|\\
&\leq 2\|{\bm{h}}\|_2^4\cdot\sum_y|{\bm{u}}|^T{\bm{B}}_y|{\bm{u}}|\\
&=2\|{\bm{h}}\|_2^2\cdot|{\bm{h}}|^T\sum_y{\bm{B}}_y|{\bm{h}}|\\
&=2\|{\bm{h}}\|_2^2\cdot|{\bm{h}}|^T({\bm{1}}_\textnormal{mat}+{\bm{I}})|{\bm{h}}|\\
&=2\|{\bm{h}}\|_2^2\cdot(\|{\bm{h}}\|_1^2+\|{\bm{h}}\|_2^2)\\
&\leq 4\|{\bm{h}}\|_2^2\cdot\|{\bm{h}}\|_1^2\,,
\end{split}
\end{align}
where ${\bm{u}}=\frac{1}{\|{\bm{h}}\|_2}{\bm{h}}$, ${\bm{1}}_\textnormal{mat}$ is a matrix of all $1$s and ${\bm{I}}$ is the identity matrix. The first inequality in \eqref{eq:ub_hDh} is obtained by $|{\bm{u}}^T{\bm{B}}_y{\bm{u}}|=|\langle{\bm{u}},{\bm{B}}_y{\bm{u}}\rangle|\leq\|{\bm{u}}\|_2\|{\bm{B}}_y{\bm{u}}\|_2\leq\|{\bm{B}}_y\|_{op}$ and hence $\frac{|{\bm{u}}^T{\bm{B}}_y{\bm{u}}|}{\|{\bm{B}}_y\|_{op}}\leq 1$; the second inequality is obtained by $|{\bm{u}}^\textrm{T}{\bm{B}}_y{\bm{u}}|=|\sum_{ij}A_y(i,j)u_iu_j|\leq\sum_{ij}A_y(i,j)|u_i||u_j|=|{\bm{u}}|^\textrm{T}{\bm{B}}_y|{\bm{u}}|$. If we choose the operator norm $\|\cdot\|_{op}$ to be the Euclidean norm, then $\|{\bm{B}}_y\|_{op}=\sigma_{\max}({\bm{B}}_y)\leq 2$; the last inequality is obtained via $\|{\bm{h}}\|_2\leq\|{\bm{h}}\|_1$.

The right-hand side of \eqref{eq:second_term_lb1_two} can be low-bounded as:
\begin{align}
\label{eq:lb_hdx}
\begin{split}
\sum_y\left({\bm{h}}^T{\bm{B}}_y{\bm{x}}\right)^2&=\|{\bm{h}}\|_1^2\cdot{\bm{v}}^\textrm{T}\left(\sum_y{\bm{B}}_y{\bm{x}}{{\bm{x}}}^\textrm{T}{\bm{B}}_y^\textrm{T}\right){\bm{v}}=\|{\bm{h}}\|_1^2\cdot{\bm{v}}^\textrm{T}{\bm{E}}{\bm{v}}\\
&\geq\|{\bm{h}}\|_1^2\cdot\lambda_{\bm{E}}\,,
\end{split}
\end{align}
where ${\bm{v}}=\frac{1}{\|{\bm{h}}\|_1}{\bm{h}}$, ${\bm{E}}=\sum_{y=0}^{M-1}{\bm{B}}_y{\bm{x}}{{\bm{x}}}^T{\bm{B}}_y^T$ and $\lambda_{\bm{E}}>0$ can be computed using Lemma \ref{LEMMA:E_MIN}. Combining \eqref{eq:second_term_lb1_two}, \eqref{eq:ub_hDh} and \eqref{eq:lb_hdx}, we can see that as long as the following \eqref{eq:h_bd} holds, \eqref{eq:second_term_lb1_two} will also hold.
\begin{align}
\label{eq:h_bd}
\|{\bm{h}}\|_2<\tau=\left(2-\frac{1}{q}\right)\cdot\sqrt{\frac{\lambda_{\bm{E}}}{4}}\,.
\end{align}
The above \eqref{eq:h_bd} guarantees that \eqref{eq:second_term_lb1} is always greater than $0$. We then have
\begin{align}
\label{eq:lb_gradient_align}
-\sqrt{\frac{1}{MK^2}\sum_y\left(({\bm{z}}_t-{\bm{x}})^\textrm{T}{\bm{B}}_y{\bm{x}}\right)^2}>-q\sqrt{4f({\bm{z}}_t)}\,.
\end{align}
Plug the above \eqref{eq:lb_gradient_align} into \eqref{eq:second_term}. We have:
\begin{align}
\label{eq:second_term_lb}
\langle{\bm{z}}_t-{\bm{x}},\, \nabla f({\bm{z}}_t)\rangle > 4(1-q)f({\bm{z}}_t)\,.
\end{align}
Plug \eqref{eq:first_term_lb}, \eqref{eq:h_bd} and \eqref{eq:second_term_lb} into \eqref{eq:diff}. We have:
\begin{align}
\label{eq:diff2}
\begin{split}
&\left\|{\bm{z}}_t-\eta\nabla f({\bm{z}}_t)-{\bm{x}} \right\|^2_2-\mu\|{\bm{z}}_t-{\bm{x}}\|^2_2\\
<&\frac{16}{K^2}\|{\bm{z}}_t\|_2^2f({\bm{z}}_t)\cdot\eta^2-8(1-q)f({\bm{z}}_t)\cdot\eta+(1-\mu)\tau^2\\
=&\frac{16}{K^2}\|{\bm{z}}_t\|_2^2f({\bm{z}}_t) \cdot \left(\left(\eta-\frac{(1-q)K^2}{4\|{\bm{z}}_t\|_2^2}\right)^2-\frac{(1-q)^2K^4}{16\|{\bm{z}}_t\|_2^4}+\frac{(1-\mu)K^2\tau^2}{16\|{\bm{z}}_t\|_2^2f({\bm{z}}_t)}\right)\,.
\end{split}
\end{align}
In order to make \eqref{eq:diff2} strictly less than $0$, the following should hold:
\begin{align}
\label{eq:condition1}
\left(\eta-\frac{(1-q)K^2}{4\|{\bm{z}}_t\|_2^2}\right)^2 < \frac{(1-q)^2K^4}{16\|{\bm{z}}_t\|_2^4}-\frac{(1-\mu)K^2\tau^2}{16\|{\bm{z}}_t\|_2^2f({\bm{z}}_t)}\,.
\end{align}
The right hand side of \eqref{eq:condition1} should be strictly greater than $0$ so that a valid $\eta$ can be obtained. This requires
\begin{align}
\label{eq:mu_lb}
\mu>1-\frac{(1-q)^2K^2}{\|{\bm{z}}_t\|_2^2}\frac{f({\bm{z}}_t)}{\tau^2}\,.
\end{align}
We also need $\mu\in(0,1)$ to ensure convergence towards the global optimum ${\bm{x}}$. Hence
\begin{align}
\max\left(0,\ 1-\frac{(1-q)^2K^2}{\|{\bm{z}}_t\|_2^2}\frac{f({\bm{z}}_t)}{\tau^2}\right)<\mu<1\,.
\end{align}
The step size $\eta$ is then:
\begin{align}
\frac{(1-q)K^2}{4\|{\bm{z}}_t\|_2^2}-\nu<\eta<\frac{(1-q)K^2}{4\|{\bm{z}}_t\|_2^2}+\nu\,,
\end{align}
where $\nu=\sqrt{\frac{(1-q)^2K^4}{16\|{\bm{z}}_t\|_2^4}-\frac{(1-\mu)K^2\tau^2}{16\|{\bm{z}}_t\|_2^2f({\bm{z}}_t)}}$.

\paragraph{Step 2:} We use ${\bm{z}}={\bm{z}}_t-\eta\nabla f({\bm{z}}_t)$ to denote the gradient descent update, and ${\bm{z}}_{t+1}=\mathscr{P}_\mathcal{S}({\bm{z}})\in\mathcal{S}$ is the projected gradient descent update. In step 1 we established conditions under which \eqref{eq:diff}$<0$, i.e.
\begin{align}
    \|{\bm{z}}-{\bm{x}}\|_2^2<\nu\|{\bm{z}}_t-{\bm{x}}\|_2^2\,.
\end{align}
Let ${\bm{s}}$ be a linear combination of ${\bm{z}}_{t+1}$ and a global optimizer ${\bm{x}}$ such that
\begin{align}
\label{eq:linear_comb}
{\bm{x}}-{\bm{z}}_{t+1} = a({\bm{z}}_{t+1}-{\bm{s}})\,,
\end{align}
where $a\in\mathbb{R}$, $a\neq 0$ is some constant. We can always find an ${\bm{s}}$ such that the following holds,
\begin{align}
\label{eq:perpendicular}
({\bm{s}}-{\bm{z}})^T({\bm{s}}-{\bm{z}}_{t+1})=0\,.
\end{align}
\begin{enumerate}[label={\arabic*.}]
\item If ${\bm{z}}={\bm{z}}_{t+1}$, then ${\bm{z}}\in\mathcal{S}$ and \eqref{eq:converge} holds.
\item If ${\bm{x}}={\bm{z}}_{t+1}$, \eqref{eq:converge} naturally holds.
\item Otherwise, we can choose $a=\frac{\|{\bm{x}}-{\bm{z}}_{t+1}\|_2^2}{\left({\bm{z}}_{t+1}-{\bm{z}}\right)^T\left({\bm{x}}-{\bm{z}}_{t+1}\right)}$. From \eqref{eq:perpendicular}, we can get:
\begin{align}
\label{eq:sum_four}
\|{\bm{s}}\|_2^2-{\bm{z}}^T{\bm{s}}-{\bm{s}}^T{\bm{z}}_{t+1} = -{\bm{z}}^T{\bm{z}}_{t+1}\,.
\end{align}
We also have
\begin{align}
\label{eq:sum_one}
\|{\bm{z}}-{\bm{z}}_{t+1}\|_2^2 &= \|{\bm{z}}\|_2^2 + \|{\bm{z}}_{t+1}\|_2^2-2{\bm{z}}^T{\bm{z}}_{t+1}\\
\label{eq:sum_two}
\|{\bm{z}}-{\bm{s}}\|_2^2&=\|{\bm{z}}\|_2^2+\|{\bm{s}}\|_2^2-2{\bm{z}}^T{\bm{s}}\\
\label{eq:sum_three}
\|{\bm{s}}-{\bm{z}}_{t+1}\|_2^2 &= \|{\bm{s}}\|_2^2 + \|{\bm{z}}_{t+1}\|_2^2 - 2{\bm{s}}^T{\bm{z}}_{t+1}\,.
\end{align}
Combining \eqref{eq:sum_four}-\eqref{eq:sum_three}, we get that
\begin{align}
\label{eq:sum_five}
\|{\bm{z}}-{\bm{z}}_{t+1}\|_2^2=\|{\bm{z}}-{\bm{s}}\|_2^2+\|{\bm{s}}-{\bm{z}}_{t+1}\|_2^2\,.
\end{align}
Using \eqref{eq:linear_comb}, we have
\begin{align}
\label{eq:sz_xs}
    {\bm{s}}-{\bm{z}}_{t+1} = \frac{1}{a+1}({\bm{s}}-{\bm{x}})\,.
\end{align}
Plug \eqref{eq:sz_xs} into \eqref{eq:perpendicular}. We have
\begin{align}
    ({\bm{s}}-{\bm{z}})^T({\bm{s}}-{\bm{x}})=0\,.
\end{align}
Similarly, we can get that
\begin{align}
\label{eq:sum_six}
\|{\bm{z}}-{\bm{x}}\|_2^2=\|{\bm{z}}-{\bm{s}}\|_2^2+\|{\bm{s}}-{\bm{x}}\|_2^2\,.
\end{align}

\begin{enumerate}[label*={\arabic*)}]
\item If ${\bm{s}}\in\mathcal{S}$, since ${\bm{z}}_{t+1}$ is the projection of ${\bm{z}}$ in $\mathcal{S}$, we have $\|{\bm{z}}-{\bm{z}}_{t+1}\|_2^2\leq\|{\bm{z}}-{\bm{s}}\|_2^2$. Using \eqref{eq:sum_five}, we have:
\begin{align}
\|{\bm{s}}-{\bm{z}}_{t+1}\|_2^2=0\,.
\end{align}
Hence ${\bm{s}}$ and ${\bm{z}}_{t+1}$ is the same point. From \eqref{eq:sum_six}, we can get:
\begin{align}
\|{\bm{z}}-{\bm{x}}\|_2^2=\|{\bm{z}}-{\bm{z}}_{t+1}\|_2^2+\|{\bm{z}}_{t+1}-{\bm{x}}\|_2^2\,.
\end{align}
Since ${\bm{z}}\notin\mathcal{S}$, we have $\|{\bm{z}}-{\bm{z}}_{t+1}\|_2^2>0$. Hence $\|{\bm{z}}-{\bm{x}}\|_2^2>\|{\bm{z}}_{t+1}-{\bm{x}}\|_2^2$.
\item If ${\bm{s}}\notin\mathcal{S}$, we have:
\begin{align}
\label{eq:sum_seven}
\begin{split}
\|{\bm{s}}-{\bm{x}}\|_2^2&=\|{\bm{s}}-{\bm{z}}_{t+1}+{\bm{z}}_{t+1}-{\bm{x}}\|_2^2\\
&=\|{\bm{s}}-{\bm{z}}_{t+1}\|_2^2+\|{\bm{z}}_{t+1}-{\bm{x}}\|_2^2+2\left({\bm{s}}-{\bm{z}}_{t+1}\right)^\textrm{T}\left({\bm{z}}_{t+1}-{\bm{x}}\right)\,.
\end{split}
\end{align}
\begin{itemize}
\item If $a\in[0, \infty)$, from \eqref{eq:linear_comb}, we have $\left({\bm{s}}-{\bm{z}}_{t+1}\right)^\textrm{T}\left({\bm{z}}_{t+1}-{\bm{x}}\right)\geq 0$. From \eqref{eq:sum_seven}, we have $\|{\bm{s}}-{\bm{x}}\|_2^2\geq\|{\bm{z}}_{t+1}-{\bm{x}}\|_2^2$. Using \eqref{eq:sum_six}, we have $\|{\bm{z}}-{\bm{x}}\|_2^2\geq\|{\bm{z}}_{t+1}-{\bm{x}}\|_2^2$.
\item If $a\in(-1,0)$, from \eqref{eq:linear_comb}, we have ${\bm{z}}_{t+1}-{\bm{x}}=\frac{a}{-1-a}({\bm{x}}-{\bm{s}})$. Since $\frac{a}{-1-a}>0$, $({\bm{z}}_{t+1}-{\bm{x}})^T({\bm{x}}-{\bm{s}})>0$. We then have:
\begin{align}
\begin{split}
\|{\bm{z}}_{t+1}-{\bm{s}}\|_2^2&=\|{\bm{z}}_{t+1}-{\bm{x}}+{\bm{x}}-{\bm{s}}\|_2^2\\
&=\|{\bm{z}}_{t+1}-{\bm{x}}\|_2^2+\|{\bm{x}}-{\bm{s}}\|_2^2+2({\bm{z}}_{t+1}-{\bm{x}})^T({\bm{x}}-{\bm{s}})\\
&>\|{\bm{x}}-{\bm{s}}\|_2^2\,.
\end{split}
\end{align}
Using \eqref{eq:sum_five} and \eqref{eq:sum_six}, we have:
\begin{align}
\|{\bm{z}}-{\bm{z}}_{t+1}\|_2^2>\|{\bm{z}}-{\bm{x}}\|_2^2\,.
\end{align}
This is in contradiction with the assumption that ${\bm{z}}_{t+1}$ is the projection of ${\bm{z}}$ in $\mathcal{S}$ so that ${\bm{z}}_{t+1}$ is closest point in $\mathcal{S}$ to ${\bm{z}}$ in terms of $l_2$ norm: $\|{\bm{z}}-{\bm{z}}_{t+1}\|_2^2\leq\|{\bm{z}}-{\bm{x}}\|_2^2$, hence $a\notin(-1,0)$.
\item If $a\in(-\infty, -1]$, from \eqref{eq:linear_comb}, we have ${\bm{s}}=-\frac{1}{a}{\bm{x}}+(1+\frac{1}{a}){\bm{z}}_{t+1}$. Since $-\frac{1}{a}\in(0,1]$, ${\bm{s}}\in\mathcal{S}$. This is in contradiction with the assumption ${\bm{s}}\notin\mathcal{S}$, hence $a\notin(-\infty,1]$
\end{itemize}
\end{enumerate}
In summary, we have
\begin{align}
\|{\bm{z}}_{t+1}-{\bm{x}}\|_2^2\leq\|{\bm{z}}-{\bm{x}}\|_2^2<\mu\|{\bm{z}}_t-{\bm{x}}\|_2^2\,.
\end{align}
\end{enumerate}
\vspace{0.2cm}

\section{Introduction}
\label{sec:intro}
\input{1_intro.tex}

\section{The Noisy Turnpike Problem}
\label{sec:main_turnpike}
\input{2_propose_turnpike.tex}

\section{The Noisy Beltway Problem}
\label{sec:main_beltway}
\input{3_propose_beltway.tex}

\section{Experimental Results}
\label{sec:exp}
\input{4_experiments.tex}

\section{Conclusion}
\label{sec:con}
\input{5_conclusion.tex}

\bibliographystyle{IEEEbib}

Data Structures and Algorithms (cs.DS); Information Theory (cs.IT); Machine Learning (cs.LG)