All RNNs trained were design to have a single layer with 8 hidden units. RNN weights were adjusted using gradients computed via back-propagation through time (BPTT) (with a look-back that extended 50 steps). Gradients were hard-clipped to have a maximum magnitude of 15 in order to ensure that the gradients did not explode explode. Adam [1]} was used to update the weights given the gradients using an initial learning rate \(2e-3\) . A patience schedule was used to adjust the learning rate – if a gain was not observed in validation within the last three times it was checked, the learning rate was halved.
We ran our experimental simulations 10 times, i.e., each run used a unique seed, and we report the mean and best performance accuracy for each model. We optimized all networks over the course of 30 epochs and report the final training and testing performance for each model.
