Specifically, we apply Theorem REF  for \(C_1=C_2=C_3=1\) . Let \(\theta (0):=\text{vec}\left(W(0), A(0), b(0)\right)\) , where we choose \(W(0)\)  such that \(\sigma _{\min }[\Phi (0)] > 0\) , \(A(0)_{ij}\ge 0\)  with \(\Vert A(0)\Vert =\Vert W(0)\Vert \) , and \(b(0)=0\) . For \(\sigma _{\min }[\Phi (0)]>0\) , a concrete example is that \(W(0)\)  is chosen according to LeCun's initialization (e.g., see [1]},[2]}). It is easy to come up with examples for which \(A(0)_{ij}\ge 0\)  and \(\Vert A(0)\Vert =\mathcal {O}\left(\Vert W(0)\Vert \right)\) . For example, \(A(0):=\Vert W(0)\Vert I_m\) . Thus, for simplicity, we assume \(\Vert A(0)\Vert =\Vert W(0)\Vert \) , then \(\lambda _2 = \lambda _1\) . Since \(b(0)=0\) , we have \(\lambda _3=1\) , and \(\Vert \hat{y}(0)-y\Vert =\Vert y\Vert \) . Next, we can choose \(\gamma >0\)  small enough for which \(\gamma _0=1/2\) . It follows from Lemma REF  that the initial conditions (REF )-() become
\(\left(\frac{2}{3}\sigma _{\min }[\Phi (0)]\right)^2\ge &16\lambda _1\Vert X\Vert _F\Vert y\Vert \\\left(\frac{2}{3}\sigma _{\min }[\Phi (0)]\right)^3\ge &128\Vert X\Vert _F^2\Vert y\Vert \\\left(\frac{2}{3}\sigma _{\min }[\Phi (0)]\right)^2\ge &128\Vert X\Vert _F^2.\) 
