Tasks. We consider two problems. The first one is a logistic regression problem on the MNIST dataset [1]} with the cost function \(f(\mathbf {x}_1,\cdots ,\mathbf {x}_{10})=-\frac{1}{m}\sum _{i=1}^{m}\sum _{j=1}^{10}l_j^i\log \big (\frac{u_j}{\sum _{j^{\prime }=1}^{10}u_{j^{\prime }}}\big )+\frac{r}{2}\Vert \mathbf {x}_j\Vert ^2\) , where \(m=60000,l_j^i\in \lbrace 0,1\rbrace \) , \(\mathbf {h}^i\in \mathbb {R}^{784}\) , \(u_j=\exp ( \mathbf {x}_j^\mathsf {T}\mathbf {h}^i)\) , and \(r=0.1\) . We sort the samples by their labels and then evenly divide them into \(n\)  parts to create heterogeneous local datasets, and each node has exclusive access to one of them. This setting is more difficult than the random partition [2]}. The second problem is an average consensus problem where nodes are supposed to find the mean \(\bar{\mathbf {x}}=\sum _{i=1}^n{\mathbf {x}}_i\) , and \({\mathbf {x}}_i\in \mathbb {R}^{10000}\)  are randomly generated from a normal distribution.
