Main contribution. In this paper, we propose to establish the global convergence
results for implicit neural networks with the nonlinear
ReLU activation function and regular output layer. Specifically, we provide sufficient conditions for the initialization under which GF and GD are guaranteed to converge to a global minimum at a linear rate. Then, we show that all these initial assumptions can be satisfied by a subset of initialization where the network has linear width in the sample size, i.e., \(m=\Omega (N)\) . For popular random initialization, we show that these initial conditions are satisfied with a high probability (w.h.p.) if the width of a network is quadratic of the sample size, i.e., \(m=\Omega (N^2)\) . Although these results with similar order of overparameterization have been obtained for finite-depth feed-forward neural networks [1]}, [2]}, [3]}, it is worth noting that this is the first time, to our best knowledge, that such results are provided for ReLU-activated implicit networks, which could have infinitely many layers.
