We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are unevenly distributed over an extremely large number of nodes. The goal is to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of the utmost importance and minimizing the number of rounds of communication is the principal goal. A motivating example arises when we keep the training data locally on users' mobile devices instead of logging it to a data center for training. In federated optimziation, the devices are used as compute nodes performing computation on their local data in order to update a global model. We suppose that we have extremely large number of devices in the network --- as many as the number of users of a given service, each of which has only a tiny fraction of the total data available. In particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, it is reasonable to assume that no device has a representative sample of the overall distribution. We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results for sparse convex problems. This work also sets a path for future research needed in the context of optimization.
Recently, lower and upper bounds for complexity of stochastic methods on problems of the form were recently obtained in @cite_102 .
@cite_102
2953275438 2790417304 2096840748 2071020383
Gimbert and Horn gave an algorithm for solving simple stochastic games with running time O(r! n) where n is the number of positions of the simple stochastic game and r is the number of its coin toss positions. pointed out that a variant of strategy iteration can be implemented to solve this problem in time 4^r r^ O(1) n^ O(1) . In this paper, we show that an algorithm combining value iteration with retrograde analysis achieves a time bound of O(r 2^r (r log r + n)), thus improving both time bounds. While the algorithm is simple, the analysis leading to this time bound is involved, using techniques of extremal combinatorics to identify worst case instances for the algorithm. We consider an algorithm that successively samples and minimizes stochastic models of the objective function. We show that under weak-convexity and Lipschitz conditions, the algorithm drives the expected norm of the gradient of the Moreau envelope to zero at the rate @math . Our result yields new complexity guarantees for the stochastic proximal point algorithm on weakly convex problems and for the stochastic prox-linear algorithm for minimizing compositions of convex functions with smooth maps. Moreover, our result also recovers the recently obtained complexity estimate for the stochastic proximal subgradient method on weakly convex problems. Relative to the large literature on upper bounds on complexity of convex optimization, lesser attention has been paid to the fundamental hardn4516420ess of these problems. Given the extensive use of convex optimization in machine learning and statistics, gaining an understanding of these complexity-theoretic issues is important. In this paper, we study the complexity of stochastic convex optimization in an oracle model of computation. We introduce a new notion of discrepancy between functions, and use it to reduce problems of stochastic convex optimization to statistical parameter estimation, which can be lower bounded using information-theoretic methods. Using this approach, we improve upon known results and obtain tight minimax complexity estimates for various function classes. We consider the following norm ∥ƒ for Boolean functions ƒ: 0, 1 n × 0, 1 n → 0, 1 : ∥ƒ|| = max ¦M(ƒ∗)v¦, v ϵ R2n, ¦v¦ = 1 . Here, M(ƒ∗) denotes the 2n × 2n-matrix obtained by setting M(ƒ∗)x,y=1 if ƒ(x,y)=1−1 if ƒ(x,y)=0. Further, we refer to the usual product of real matrices and vectors and denote by ¦v¦ the Euclidean norm of real vectors v. In this paper it will be shown by geometric arguments that for each function ƒ: 0,1 n × 0,1 n → 0,1 the following is true. - The length of each probabilistic communication protocol which computes ƒ with error bounded by 12 − 1s, s ϵ N, can be estimated from below by 14(n − log2∥ƒ||−log2 √s−2). - The number of edges of any threshold circuit of depth two computing ƒ cannot be smaller than 2n−1‖ƒ‖ These results yield better lower bounds on probabilistic communication complexity as well as on the complexity of threshold circuits of depth two. Further, characterizing ∥ƒ∥ by the eigenvalues of M(ƒ∗) we obtain a method to construct iteratively functions which are hard to compute in the above models. This implies lower bounds for decision problems that previous probabilistic techniques from Yao (1983), Halstenberg and Reischuk (1988) and (1987) cannot be applied to.