@inproceedings{
li2024resourceaware,
title={Resource-Aware Federated Self-Supervised Learning with Global Class Representations},
author={Mingyi Li and Xiao Zhang and Qi Wang and Tengfei LIU and Ruofan Wu and Weiqiang Wang and Fuzhen Zhuang and Hui Xiong and Dongxiao Yu},
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
year={2024},
url={https://openreview.net/forum?id=Of4iNAIUSe}
}
Due to the heterogeneous architectures and class skew, the global representation models training in resource-adaptive federated self-supervised learning face with tricky challenges: $\textit{deviated representation abilities}$ and $\textit{inconsistent representation spaces}$. 
In this work, we are the first to propose a multi-teacher knowledge distillation framework, namely $\textit{FedMKD}$, to learn global representations with whole class knowledge from heterogeneous clients even under extreme class skew. Firstly, the adaptive knowledge integration mechanism is designed to learn better representations from all heterogeneous models with deviated representation abilities. Then the weighted combination of the self-supervised loss and the distillation loss can support the global model to encode all classes from clients into a unified space. Besides, the global knowledge anchored alignment module can make the local representation spaces close to the global spaces, which further improves the representation abilities of local ones. Finally, extensive experiments conducted on two datasets demonstrate the effectiveness of $\textit{FedMKD}$ which outperforms state-of-the-art baselines 4.78\% under linear evaluation on average.
Resource-Aware Federated Self-Supervised Learning with Global Class Representations
Mingyi Li Xiao Zhang Qi Wang Tengfei LIU Ruofan Wu Weiqiang Wang Fuzhen Zhuang Hui Xiong Dongxiao Yu