We follow the weight initialization technique mentioned in [1]} to use a trivial N (0, 1)
initialization and then explicitly scale the weights at run-time [2]}. We set \(w^{\prime }_i\)  = \(w_i/c\) ,
where \(w_i\)  are the weights and c is the per-layer normalization constant from He's initializer. This ensures that the dynamic range, and thus the learning speed, is the same for all weights.
