where \(\mathbf {y}(n)\)  is the output vector and \(W^{\rm out}\)  is the weight matrix in the readout. In supervised learning, this weight matrix is trained to minimize the difference between the network output and the desired output for a certain time period (see Sec. 8 of [1]} for details). The performance of the ESN depends on the design of the RNN-based reservoir. In order to approximate a teacher output signal, the RNN-based reservoir must have the echo state property, whereby it asymptotically eliminates any information from the initial conditions [2]}. It is empirically observed that the echo state property is obtained for any input if the spectral radius (i.e. the maximum eigenvalue of \(W\) ) is adjusted to be smaller than unity. A practical guide for reservoir design in ESNs can be found in [3]}. The echo state property is essentially the same as the fading memory property [4]}, [5]}. An input/output system (or a filter) for temporal signals has fading memory when the outputs corresponding to inputs that are close in the recent past are close even if those inputs are very different in the distant past. Recently, it was theoretically shown that ESNs have universal approximation ability in terms of discrete-time fading memory filters with uniformly bounded inputs defined on negative infinite times [6]}.
