We show that conditions (REF )-() are satisfied under similar random initialization as LeCun's Initialization [1]}, Xavier Initialization [2]}, and He Initialization [3]}, provided a stronger condition \(m=\Omega (N^2)\) .
For simplicity, we assume \(\Vert x_i\Vert =1\)  and \(\left|y_i\right|=\mathcal {O}\left(1\right)\)  for all \(i\in [N]\) . Then \(\Vert X\Vert _F=\sqrt{N}\)  and \(\Vert y\Vert =\mathcal {O}\left(\sqrt{N}\right)\) . Consider initialization
\(b(0)_{i}\overset{i.i.d.}{\sim } \mathcal {N}(0,1/m),\quad A(0)_{ij}\overset{i.i.d.}{\sim } \left|\mathcal {N}\right|(0,1),\quad W(0)_{ij}\overset{i.i.d.}{\sim } \mathcal {N}(0,1),\) 
