Small neural network models typically contain 2 to 4 hidden layers with a moderate number of hidden units per layer [1]}, [2]}, [3]}. Hence, small DNN models can be trained by standard gradient descent [4]}, where all the available training samples are used for model updates. However, there has been a consistent increase in the difficulty of learning problems tackled over time; recent benchmarking datasets typically contain thousands or several millions of training samples, e.g, Places [5]}, ImageNet [6]} and UMD faces [7]} datasets. Thus, there has been a consistent increase in the depth and number of DNN model parameters ever since. Subsequently, mini-batch gradient descent, which uses a specified portion of available training samples for model updates every iteration, has been favoured for training large DNN models, in view of fast training time and effective utilization of available computing resources. Therefore, fast training of large DNNs have become an important research problem.
