We trained both the analytical and the stochastic models, as well as all baselines against which we compare,
using Adam [1]}. We used \(\alpha =0.001,\beta _1=0.9, \beta _2=0.999\)  for one hidden layer networks,
and \(\alpha =0.001\)  for the networks with more hidden layers. We initialize all networks parameters using
[2]}.
Due to the large computational complexity of the analytical approach we set the mini-batch size \(m\)  to five.
For the stochastic model, as well as for all the baseline models, we set the mini-batch size to 20.
For the analytical model we set the maximum number of iterations to 5000.
For the stochastic model we set the maximum number of iterations to 10000 for the one layer networks and
to 20000 for networks with more layers. We used early stopping where we keep 20% of the training data as
the validation set. Every five parameter updates we compute the validation error. Training terminates either
if we reach the maximum iteration number or the validation error keeps increasing more than ten times in a row.
