We use Xavier initialization [1]} with gain of 1 to initialize the all weight matrices. For shallow two-layer networks that will not enter the EoS regime if using completely random initialization, we will asymmetrically re-scale the layers after random initialization by multiplying a constant to all entries of the same layer. When we present results for the re-scaled experiments, we will state the re-scale factor.
