Initialization From the optimization perspective, initializations are usually based on the "edge of chaos" idea such that variance of features and gradients should be preserved in the forward and backward pass [1]}, [2]}, [3]}, [4]}, [5]}, or based on analyzing Jacobians and/or Hessians [6]}. From the generalization perspective, it was shown that large initializations overfit data easily but result in poor generalization [7]}, [8]}, which agrees with our LU mechanism.
