The convergence problem of gradient flow and gradient descent for standard deep networks has been studied recently [1]}, [2]}, [3]}, [4]}, [5]}, [6]}. To show the convergence, the most common strategy is NTK method [7]}, where it can be shown that dynamic of the prediction is governed by a gram matrix whose smallest eigenvalues is kept strictly positive during training. The previous work of [1]}, [2]}, [3]}, [5]} requires all hidden layers have large widths since their analyses rely on studying various quantities related to the changes in the activation patterns during training. In [4]}, the authors claim one wide hidden layer is enough if the network follows a pyramidal topology and the derivative of activation is lower bounded. Moreover, they prove that linear width is enough for overparameterization, while previous works need a width of at least \(\Omega (N^8)\) . The most recent study [6]} further confirms linear width suffices global convergence for ReLU activated networks by focusing the analysis on the last hidden layer so that the pyramidal topology is not necessary. However, none of the previous works can be applied to implicit networks directly since implicit networks have infinitely many layers, and equilibrium equations may not be well-posed during training.
