In general, the step size or learning rate \(\eta \)  as a hyper-parameter is significant on both training and test performance. However, the choice of \(\eta \)  mostly likely is based on the practical experiments. Based on some theoretical analyses, \(\eta = \mathcal {O}\left(L^{-1}\right)\)  is one of the widely obtained result, where \(L\)  is the Lipschitz constant by assuming the gradient of neural network is Lipschitz continuous. Unfortunately, the value of \(L\)  is generally unknown or relatively large [1]}. In Section , we show \(\eta =\mathcal {O}\left(N^{-1}\right)\)  is enough to guarantee the convergence of the gradient descent, which is much larger than many previous results \(\eta =\mathcal {O}\left(N^{-2}\right)\)  [2]}, [3]}, [4]}, even the implicit neural network could have infinitely many layers. Thus, we also construct a series of numerical experiments to justify the choice of step size \(\eta \) . With almost the same setup as before, we set the values for step size \(\eta \)  from \(\lbrace 10^{-2}, 10^{-3}, 10^{-4}\rbrace \) . The corresponding train losses and test losses are illustrated in Figure REF . The first sub-figure in Figure REF  shows that the objective value does not consistently decrease if we choose the step size \(\eta \)  larger than the suggested value in Theorem REF . Accordingly, the second sub-figure in Figure REF  shows instability of the test loss for large step size. In addition, the operator norm of \(\gamma A(k)\)  raises fast to a relatively large value, which probably causes divergence of the forward propagation. Therefore, choosing step size \(\eta \)  larger than \(N^{-1}\)  is probably causing the divergence of the forward propagation and the entire training process. This experimentally indicates the result of the step size \(\eta \)  in Theorem REF  is tight.
