Recently, implicit neural networks attracts increasing attention in the machine learning community, which have achieved competitive or dominated performances of traditional neural networks in various domains such as sequence modeling [1]} with significantly less usage of computational resources [2]}, [3]}, [4]}. In implicit neural networks, the feature vectors are not created recursively as traditional neural networks but provided implicitly through a solution of an equilibrium equation. Implicit neural networks generalize the recursive rules of many commonly used neural network architectures such as feed-forward, convolution, residual, and recurrent networks [1]}, [6]}, [7]}.
However, the theoretical understanding of when and how a simple first-order method such as gradient descent (GD) works for these implicit models is limited, though this type of convergence problem has been well studied in standard feed-forward networks [8]}, [9]}, [10]}, [11]}, [12]}, [13]}, [14]}.
Since implicit networks can have infinitely many layers, the equilibrium equation is not necessarily well-posed as it may admit zero or multiple solutions during training. For example, [15]}, [1]}, [17]}, [18]} all observe instability of forward propagations in implicit models. Specifically, the number of iterations required for forward propagation to find equilibrium points grows with training epochs. Forward propagation likely becomes divergent as training goes longer. A line of recent works make efforts to handle this well-posedness challenge. For example, [6]} reformulates the training problem in a so-called Fenchel divergence formulation and use the projected gradient descent method to solve the relaxed optimization problem; [20]} formulates a splitting problem for the forward propagation of implicit neural networks and use a proximal operation to find the fixed point; [17]} proposes a special regularization to ensure the well-posedness. However, none of these works can theoretically ensure the convergence of a gradient-based method.
