For learning the parameters of these NNs, the gradient of this loss with respect to all the parameters, \(\theta \) , must be calculated. This gradient can either be calculated by backpropagating through the solver with automatic differentiation or by solving an adjoint problem backwards in time [1]}. When backpropagating all of the data must be stored at each time-step, which becomes memory intensive when the prediction horizon becomes large. For our trials we do not consider training data spaced apart far enough in time where this becomes a problem, and empirically we found using the adjoint method took longer, so we use backpropagation for training.
