Thus, to lower bound the singular values of matrices \(M(t)\)  and \(\Pi (t)\) , one must make extra assumptions on the data sample \(X\)  and activation \(\sigma \) . An example method is introduced in [1]}, where the neural network has to follow a pyramidal structure and the activation function has to be sufficiently smooth. This method does not hold for ReLU activation due to the non-smoothness. In contrast, we adopt the method proposed by [2]} for finite-depth ReLU neural network, where their analysis focuses on the evolution of the last layer of the network, that is, \(Z(t)\)  in the implicit neural network. With appropriate assumptions on the initial conditions, we can establish the global convergence result for the gradient flow in Theorem REF , and the entire proof is provided in Appendix REF .
