In Federated Learning, periodic full model aggregation is the most common approach for aggregating local models across clients.
Many Federated Learning algorithms, such as FedAvg ([1]}), FedProx ([2]}), FedNova ([3]}), and SCAFFOLD ([4]}), assume the underlying periodic full aggregation scheme.
However, it has been observed that the magnitude of gradients can be significantly different across the layers of neural networks ([5]}).
That is, all the layers can have a different degree of model discrepancy.
The periodic full aggregation scheme does not consider such a difference and synchronizes the entire model parameters at once.
Aggregating the parameters that are similar across all the clients does not make meaningful training progress while increasing the communication cost.
Considering the limited network bandwidth in usual Federated Learning environments, such an inefficient network bandwidth consumption can significantly harm the scalability of Federated Learning applications.
