I built out a fairly standard backpropagation algorithm and just the process of forward propagating through a 5 layer x 5 nodes network using a data set of 10000 observations of 39 variables takes almost 5 minutes for one iteration. Do neural networks typically take multiple hours to train using data sets this size? My initial data set was 10x as long, but I couldn't wait an hour just for one forward pass to be completed. 
Neural Network training time?
neural network