In recent yeas, a great number of researches have been devoted to dealing with large-scale statistical optimization problems. For example, the distributed computing methods, based on the divide-and-conquer strategy from computer science literature, become very popular in the statistics community.
Many statistical learning problems have been investigated on a distributed system, such as generalized linear models [1]}, [2]}, [3]}, quantile regression [4]}, [5]}, kernel ridge regression [6]}, [7]}, support vector machine [8]}, principal component analysis [9]}, [10]}, and references therein.
Clearly, the distributed computing frameworks reduce the computational burden of one single machine by employing a computer cluster.
However, it still replies on a distributed architecture for implementation, which can be expensive for many practical problems.
With limited computational resources, it is desirable
to conduct computation with a single computer and feasible optimization algorithms.
In this regard, stochastic optimization methods, especially stochastic gradient descent (SGD) and its derivatives, play an important role.
