Asymptotic Stability in Reservoir Computing
Reservoir Computing is a class of Recurrent Neural Networks with internal
weights fixed at random. Stability relates to the sensitivity of the network
state to perturbations. It is an important property in Reservoir Computing as
it directly impacts performance. In practice, it is desirable to stay in a
stable regime, where the effect of perturbations does not explode
exponentially, but also close to the chaotic frontier where reservoir dynamics
are rich. Open questions remain today regarding input regularization and
discontinuous activation functions. In this work, we use the recurrent kernel
limit to draw new insights on stability in reservoir computing. This limit
corresponds to large reservoir sizes, and it already becomes relevant for
reservoirs with a few hundred neurons. We obtain a quantitative
characterization of the frontier between stability and chaos, which can greatly
benefit hyperparameter tuning. In a broader sense, our results contribute to
understanding the complex dynamics of Recurrent Neural Networks.