I'm training a 2-layer, 1024 node, dropout of 0.5 RNN over natural text. Specifically, I'm using karpathy's char-rnn which I found to work quite well for most of my use cases. Sometimes however, late in the training, my errors drastically increase, and I can't figure out why this is happening.The question is specifically asking why am I observing the drastic increase in error, and how this is handled in practice. What can be done to prevent this besides taking the last-known good model?
What causes the error in a RNN to increase late in training?
machine learning;rnn