For distributed computing environments, we consider the canonical machine learning problem of empirical risk minimization (ERM) with quadratic regularization, and we propose a distributed and communication-efficient Newton-type optimization method. At every iteration, each worker locally finds an Approximate NewTon (ANT) direction, and then it sends this direction to the main driver. The driver, then, averages all the ANT directions received from workers to form a Globally Improved ANT (GIANT) direction. GIANT naturally exploits the trade-offs between local computations and global communications in that more local computations result in fewer overall rounds of communications. GIANT is highly communication efficient in that, for @math -dimensional data uniformly distributed across @math workers, it has @math or @math rounds of communication and @math communication complexity per iteration. Theoretically, we show that GIANT's convergence rate is faster than first-order methods and existing distributed Newton-type methods. From a practical point-of-view, a highly beneficial feature of GIANT is that it has only one tuning parameter---the iterations of the local solver for computing an ANT direction. This is indeed in sharp contrast with many existing distributed Newton-type methods, as well as popular first-order methods, which have several tuning parameters, and whose performance can be greatly affected by the specific choices of such parameters. In this light, we empirically demonstrate the superior performance of GIANT compared with other competing methods.
In Table , we compare the communication costs with other methods for the ridge regression problem: @math . As for general convex problems, it is very hard to present the comparison in an easily understanding way. This is why we do not compare the convergence for the general convex optimization. The communication cost of GIANT has a mere logarithmic dependence on the condition number @math ; in contrast, the other methods have at least a square root dependence on @math . Even if @math is assumed to be small, say @math , which was made by @cite_31 , GIANT's bound is better than the compared methods regarding the dependence on the number of partitions, @math .
@cite_31
2755278809 2953101345 2962782955 172279872
For distributed computing environments, we consider the canonical machine learning problem of empirical risk minimization (ERM) with quadratic regularization, and we propose a distributed and communication-efficient Newton-type optimization method. At every iteration, each worker locally finds an Approximate NewTon (ANT) direction, and then it sends this direction to the main driver. The driver, then, averages all the ANT directions received from workers to form a Globally Improved ANT (GIANT) direction. GIANT naturally exploits the trade-offs between local computations and global communications in that more local computations result in fewer overall rounds of communications. GIANT is highly communication efficient in that, for @math -dimensional data uniformly distributed across @math workers, it has @math or @math rounds of communication and @math communication complexity per iteration. Theoretically, we show that GIANT's convergence rate is faster than first-order methods and existing distributed Newton-type methods. From a practical point-of-view, a highly beneficial feature of GIANT is that it has only one tuning parameter---the iterations of the local solver for computing an ANT direction. This is indeed in sharp contrast with many existing distributed Newton-type methods, as well as popular first-order methods, which have several tuning parameters, and whose performance can be greatly affected by the specific choices of such parameters. In this light, we empirically demonstrate the superior performance of GIANT compared with other competing methods. We explore the connection between dimensionality and communication cost in distributed learning problems. Specifically we study the problem of estimating the mean @math of an unknown @math dimensional gaussian distribution in the distributed setting. In this problem, the samples from the unknown distribution are distributed among @math different machines. The goal is to estimate the mean @math at the optimal minimax rate while communicating as few bits as possible. We show that in this setting, the communication cost scales linearly in the number of dimensions i.e. one needs to deal with different dimensions individually. Applying this result to previous lower bounds for one dimension in the interactive setting ZDJW13 and to our improved bounds for the simultaneous setting, we prove new lower bounds of @math and @math for the bits of communication needed to achieve the minimax squared loss, in the interactive and simultaneous settings respectively. To complement, we also demonstrate an interactive protocol achieving the minimax squared loss with @math bits of communication, which improves upon the simple simultaneous protocol by a logarithmic factor. Given the strong lower bounds in the general setting, we initiate the study of the distributed parameter estimation problems with structured parameters. Specifically, when the parameter is promised to be @math -sparse, we show a simple thresholding based protocol that achieves the same squared loss while saving a @math factor of communication. We conjecture that the tradeoff between communication and squared loss demonstrated by this protocol is essentially optimal up to logarithmic factor. We propose a new distributed algorithm based on alternating direction method of multipliers (ADMM) to minimize sum of locally known convex functions using communication over a network. This optimization problem emerges in many applications in distributed machine learning and statistical estimation. Our algorithm allows for a general choice of the communication weight matrix, which is used to combine the iterates at different nodes. We show that when functions are convex, both the objective function values and the feasibility violation converge with rate @math , where @math is the number of iterations. We then show that when functions are strongly convex and have Lipschitz continuous gradients, the sequence generated by our algorithm converges linearly to the optimal solution. In particular, an @math -optimal solution can be computed with @math iterations, where @math is the condition number of the problem. Our analysis highlights the effect of network and communication weights on the convergence rate through degrees of the nodes, the smallest nonzero eigenvalue, and operator norm of the communication matrix. Let a communication network be modelled by a directed graphG = (V,E) of n nodes and m edges. We consider aone-round two-player network pricing game, the Stackelberg ShortestPaths Tree (StackSPT) game. This is played on G, by assuming thatedges in E are partitioned into two sets: a set E F of edges with afixed positive real weight, and a set E P of edges that should bepriced by one of the two players (the leader). Given adistinguished node r ∈ V, the StackSPT game isthen as follows: the leader prices the edges in E P in such a waythat he will maximize his revenue, knowing that the other player(the follower) will build a shortest paths tree of G rooted at r,say S(r), by running a publicly available algorithm. Quitenaturally, for each edge selected in the solution, theleader’s revenue is assumed to be equal to the loaded priceof an edge, namely the product of the edge price times the numberof paths from r in S(r) that use it. First, we show that theproblem of maximizing the leader’s revenue is NP-hard as soonas |E P | = Θ(n). Then, in search of aneffective method for solving the problem when the size of E P isconstant, we focus on the basic case in which |E P| = 2, and we provide an efficient O(n 2 logn) timealgorithm. Afterwards, we generalize the approach to the case |E P| = k, and we show that it can be solved inpolynomial time whenever k = O(1).