FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning
Newton-type methods are popular in federated learning due to their fast
convergence. Still, they suffer from two main issues, namely: low communication
efficiency and low privacy due to the requirement of sending Hessian
information from clients to parameter server (PS). In this work, we introduced
a novel framework called FedNew in which there is no need to transmit Hessian
information from clients to PS, hence resolving the bottleneck to improve
communication efficiency. In addition, FedNew hides the gradient information
and results in a privacy-preserving approach compared to the existing
state-of-the-art. The core novel idea in FedNew is to introduce a two level
framework, and alternate between updating the inverse Hessian-gradient product
using only one alternating direction method of multipliers (ADMM) step and then
performing the global model update using Newton's method. Though only one ADMM
pass is used to approximate the inverse Hessian-gradient product at each
iteration, we develop a novel theoretical approach to show the converging
behavior of FedNew for convex problems. Additionally, a significant reduction
in communication overhead is achieved by utilizing stochastic quantization.
Numerical results using real datasets show the superiority of FedNew compared
to existing methods in terms of communication costs.