@inproceedings{
li2024crossmodal,
title={Cross-Modal Meta Consensus for Heterogeneous Federated Learning},
author={Shuai Li and Fan Qi and Zixin Zhang and Changsheng Xu},
booktitle={ACM Multimedia 2024},
year={2024},
url={https://openreview.net/forum?id=ufL0BQdR9L}
}
In the evolving landscape of federated learning (FL), the integration of multimodal data presents both unprecedented opportunities and significant challenges. Existing work falls short of meeting the growing demand for systems that can efficiently handle diverse tasks and modalities in rapidly changing environments. We propose a meta-learning strategy tailored for Multimodal Federated Learning (MFL) in a multitask setting, which harmonizes intra-modal and inter-modal feature spaces through the Cross-Modal Meta Consensus. This innovative approach enables seamless integration and transfer of knowledge across different data types, enhancing task personalization within modalities and facilitating effective cross-modality knowledge sharing. Additionally, we introduce Gradient Consistency-based Clustering for multimodal convergence, specifically designed to resolve conflicts at meta-initialization points arising from diverse modality distributions,  supported by theoretical guarantees. Our approach, evaluated as $M^{3}Fed$ on five federated datasets, with at most four modalities and four downstream tasks, demonstrates strong performance across diverse data distributions, affirming its effectiveness in multimodal federated learning. The code is available at https://anonymous.4open.science/r/M3Fed-44DB.
Cross-Modal Meta Consensus for Heterogeneous Federated Learning
Shuai Li Fan Qi Zixin Zhang Changsheng Xu