ADMM algorithms have been widely viewed as an optimization method that lies in between first order methods and second order methods, and many researchers have found that ADMM algorithms are well-suited for large scale statistical inference and machine learning problems [[1]}, [2]}, [3]}, [4]}]. The traditional two-block ADMM algorithm and its convergence analysis have been excessively explored [[5]}, [6]}, [7]}, [8]}, [9]}, [10]}]. However, the two-block ADMM is not well-suited for solving large scale machine learning problems, as it requires processing and factorization for a large matrix, which is both time and space consuming [[11]}, [12]}, [13]}]. Instead, variants of multi-block ADMM emerges, several prevailing multi-block ADMM algorithms include distributed ADMM [[14]}, [15]}, [16]}], cyclic ADMM [[17]}, [18]}], and randomized multi-block ADMM [[19]}, [20]}]. While distributed ADMM is known to be robust for convergence under any choice of step size parameter, many study have also found that the convergence rate of primal distributed ADMM is highly sensitive to the choice of step size, and may suffer from solve convergence once the step-size is not well-tuned [[21]}, [22]}, [23]}]. On the other hand, while cyclic ADMM can boost up the convergence rate sometimes compared with distributed ADMM, cyclic ADMM is not guaranteed to converge for all instances in theory [[24]}]. [20]} first introduce data-share across different blocks in multi-block ADMM algorithm. However, their algorithm requires a full randomized data exchange across blocks at each iteration, which may not be feasible in practice and requires more computational cost.
