The well-posedness of the forward propagation is the main challenge for any infinite-depth or implicit-depth networks. A line of recent works [1]}, [2]}, [3]}, [4]} have shown the instability of forward propagation in implicit networks, and the probability of divergent forward pass gradually raises with training epochs. Several strategies are suggested to deal with the well-posedness problem, e.g., adding constraints [5]} or regularization[2]}, reformulates the forward propagation [1]}. Unfortunately, none of them can theoretically guarantee convergence. By using an extra softmax layer on the weight matrix, [8]} proves the convergence of the gradient flow, but the result only holds for linear activation while nonlinearity is critical for learnability. To deal with well-posedness, our strategy is similar to [9]} by introducing a scalar in front of the weight matrix. The well-posedness problem is resolved since we can find a sufficiently small constant scalar. The novelty of this paper is that implicit network considered in this paper has regular output while [9]} uses a skip connection to facilitate training process. Moreover, this is the first work, to our best knowledge, that establishes the global convergence for an infinite-depth (weight tying) network with linear width.
