
where \(i\)  indicates retrieval of the \(i\) th scalar value in a vector.
Our hybrid, next-step prediction scheme was motivated by the intuition that next step prediction might facilitate a better training scheme for CFL recognition and allow the RNN to recover from previous mistakes made when iteratively recognizing the sequence. In other words, next-step prediction provides a stronger form of error correction, as was also discovered when training first order recurrent nets [1]} and also to train complex, higher-order neural models [2]}. From a model training point-of-view, having a next-step prediction scheme also provides a wealth of gradient signals, since predictions are being made and corrected at each time step, that help combat the vanishing gradient problem [3]} that plagues training RNN architectures.
This is particularly important for the CLF recognition task given that models are expected to operate and learn from very long pattern sequences.
