In fact, the many popularly used SGD algorithms in practice are based on shuffled minibatches. In other words, the entire SGD algorithm is divided into different epochs. For each epoch, the original whole dataset is randomly shuffled and then sequentially divided into different batches. Subsequently, different batches are sequentially loaded into the computer memory and then processed. Once all the batches have been successfully executed, the whole dataset might need to be reshuffled. This process accomplishes one epoch of the minibatch-based SGD algorithm. Accordingly, each sample in the whole dataset is ensured to be used exactly once for every epoch iteration. Our limited numerical experiments suggest that this approach often leads to estimators with satisfactory statistical efficiency. As a consequence, shuffled minibatch-based GD algorithms are indeed the most widely used SGD algorithms in practice. In fact, it has been implemented by many important software libraries, such as TensorFlow [1]} and PyTorch [2]}.
