Our analysis generalizes previous results on linear neural networks [1]} to a more general nonlinear setting relevant for learning problems. Our non-asymptotic results stand in contrast to the mean-field analysis [2]} which requires infinite depth. Our tractable bounds improve upon the ones found for networks over-parametrized by width [3]}, [4]}, [5]}, [6]}, [7]}, where the trained weights do not leave the lazy training regime [8]}: in our setting, the trained weights are not necessarily staying close to their initialization. A key ingredient in the proof is to study the evolution of various norms for the weights under gradient descent iterations. These estimates are provided in Lemmas REF  and REF .
