Since a massive dataset cannot be easily loaded into computer memory as a whole, it has to be processed in a batch-by-batch manner. Here, “batch” refers to a subsample of the original whole dataset. Subsequently, the important quantities (e.g., gradient and momentum) can be computed based on the batch. This approach leads to various stochastic gradient descent (SGD) methods. Here, “stochastic" refers to the batch being randomly subsampled from the original whole dataset.
Then, according to how this subsample (i.e., batch) is generated and utilized, different SGD algorithms can be developed. The history of SGD can date back to the pioneering work of [1]}.
The computational efficiency of SGD finds wide application in the large-scale optimization problems, and attracts a large number of follow-up studies [2]}, [3]}, [4]}, [5]}.
Although many existing works focus on the numerical convergence rate of the stochastic optimization algorithms, there are also a few of them that investigate the resulting estimator from a statistical viewpoint.
For example, [6]} proposed an implicit SGD procedure, and analyzed the asymptotic behavior of the estimators resulting from the standard and implicit SGD procedures. [7]} and [8]} investigated the statistical inference problem based on the averaged SGD estimator, and provided several methods to estimate the asymptotic covariance.
[9]} developed a renewable estimation procedure that extends the standard SGD and averaged implicit SGD algorithm, and further studied the asymptotic properties of the resulting estimator.
In general, the algorithms that they discussed are one-pass in nature and should be very useful for streaming data analysis. However, for non-streaming situations where the massive dataset is already on the hard drive, the one-pass type of algorithm is less frequently used in practice. Instead, multiple passes (or epochs) are more commonly used for better statistical efficiency.
