Despite its theoretical popularity, the practical implementation of GDM could be nontrivial on a massive dataset. Under a traditional setup with small sample size and simple model structure, a GDM algorithm can be easily implemented by loading the entire dataset into the computer memory as a whole. Subsequently, various gradients and momentums can be efficiently computed and processed. Unfortunately, such a common practice could be seriously challenged if the target model structure is sufficiently complex and the dataset is massive in size. For example, the ResNet50 model [1]} is a classical deep learning model with more than 25 million parameters. In this case, it would take a long time to compute the gradient and momentum needed for performing one single step of iteration. Thus, implementing GDM algorithms on such a complex model (i.e., the ResNet50 model) with such a massive dataset (i.e., the ImageNet dataset) is very challenging.
