The time to converge to the steady state of a finite Markov chain can be greatly reduced by a lifting operation, which creates a new Markov chain on an expanded state space. For a class of quadratic objectives, we show an analogous behavior where a distributed alternating direction method of multipliers (ADMM) algorithm can be seen as a lifting of gradient descent. This provides a deep insight for its faster convergence rate under optimal parameter tuning. We conjecture that this gain is always present, as opposed to the lifting of a Markov chain, which sometimes only provides a marginal speedup.
We consider a network of agents that are cooperatively solving a global unconstrained optimization problem, where the objective function is the sum of privately known local objective functions of the agents. Recent literature on distributed optimization methods for solving this problem focused on subgradient based methods, which typically converge at the rate equation, where k is the number of iterations. In this paper, we introduce a new distributed optimization algorithm based on Alternating Direction Method of Multipliers (ADMM), which is a classical method for sequentially decomposing optimization problems with coupled constraints. We show that this algorithm converges at the rate equation. The alternating direction multipliers method (ADMM) has been recently proposed as a practical and efficient algorithm for distributed computing. We discuss its applicability to the average consensus problem in this paper. By carefully relaxing ADMM augmentation coefficients we are able to analytically investigate its properties, and to propose simple and strict analytical bounds. These provide a clear indication on how to choose system parameters for optimized performance. We prove both analytically and via simulations that the proposed approach exhibits convergence speed between the best in the literature (classical and optimized solutions), while providing the most powerful resilience to noise. We describe how the powerful "Divide and Concur" algorithm for constraint satisfaction can be derived as a special case of a message-passing version of the Alternating Direction Method of Multipliers (ADMM) algorithm for convex optimization, and introduce an improved message-passing algorithm based on ADMM DC by introducing three distinct weights for messages, with "certain" and "no opinion" weights, as well as the standard weight used in ADMM DC. The "certain" messages allow our improved algorithm to implement constraint propagation as a special case, while the "no opinion" messages speed convergence for some problems by making the algorithm focus only on active constraints. We describe how our three-weight version of ADMM DC can give greatly improved performance for non-convex problems such as circle packing and solving large Sudoku puzzles, while retaining the exact performance of ADMM for convex problems. We also describe the advantages of our algorithm compared to other message-passing algorithms based upon belief propagation.