@inproceedings{
dun2023fedjets,
title={Fed{JET}s: Efficient Just-In-Time Personalization with Federated Mixture of Experts},
author={Chen Dun and Mirian Hipolito Garcia and Guoqing Zheng and Ahmed Awadallah and Robert Sim and Anastasios Kyrillidis and Dimitrios Dimitriadis},
booktitle={R0-FoMo:Robustness of Few-shot and Zero-shot Learning in Large Foundation Models},
year={2023},
url={https://openreview.net/forum?id=xGKuxPVwcL}
}
One of the goals in Federated Learning (FL) is to create personalized models that can adapt to the context of each participating client, while utilizing knowledge from a shared global model. Yet, often, personalization requires a fine-tuning step using clients' labeled data in order to achieve good performance.  This may not be feasible in scenarios where incoming clients are fresh and/or have privacy concerns. 
It, then, remains open how one can achieve just-in-time personalization in these scenarios. We propose FedJETs, a novel solution by using a Mixture-of-Experts (MoE) framework within a FL setup. Our method leverages the diversity of the clients to train specialized experts on different subsets of classes, and a gating function to route the input to the most relevant expert(s).  Our gating function harnesses the knowledge of a  pretrained model (common expert) to enhance its routing decisions on-the-fly. As a highlight, our approach can improve accuracy up to 18% in state of the art FL settings, while maintaining competitive zero-shot performance. In practice, our method can handle non-homogeneous data distributions, scale more efficiently, and improve the state-of-the-art performance on common FL benchmarks.
FedJETs: Efficient Just-In-Time Personalization with Federated Mixture of Experts
Chen Dun Mirian Hipolito Garcia Guoqing Zheng Ahmed Awadallah Robert Sim Anastasios Kyrillidis Dimitrios Dimitriadis