Under some simplified setups, a line of recent works have tried to study this convergence problem from the mathematical theory perspective. For instance, [1]} studies this problem for implicit models with linear activation function. By applying an extra softmax layer on the shared weight matrix, the well-posedness challenge is resolved. As a result, they are able to establish the global linear convergence for gradient flow (GF). Unfortunately, their results cannot be extended to nonlinear activation, especially for the nonsmooth ReLU activation, which are critical to the learnability of deep neural networks. Recently, [2]} facility the training process by using a skip connection to the output and introduces a scaling factor to scale the shared weight matrix of the network, which can show there exists a small enough scaling constant which keeps the forward propagation well-posed throughout training. To show the convergence of ReLU-activated implicit neural networks, they take a neural tangent kernel (NTK) method [3]}, where the dynamic of the network prediction is governed by a Gram matrix that remains positive definite during training. As a result, they successfully establish the global linear convergence for both GF and GD as long as the width \(m\)  of the network is quadratic in the sample size \(N\) , i.e., \(m=\Omega (N^2)\) . However, their results can only be applied to restricted range of implicit networks due to the special choice of the output layer.
