Decentralized adaptive clustering of deep nets is beneficial for client collaboration
We study the problem of training personalized deep learning models in a
decentralized peer-to-peer setting, focusing on the setting where data
distributions differ between the clients and where different clients have
different local learning tasks. We study both covariate and label shift, and
our contribution is an algorithm which for each client finds beneficial
collaborations based on a similarity estimate for the local task. Our method
does not rely on hyperparameters which are hard to estimate, such as the number
of client clusters, but rather continuously adapts to the network topology
using soft cluster assignment based on a novel adaptive gossip algorithm. We
test the proposed method in various settings where data is not independent and
identically distributed among the clients. The experimental evaluation shows
that the proposed method performs better than previous state-of-the-art
algorithms for this problem setting, and handles situations well where previous
methods fail.