Federated Latent Class Regression for Hierarchical Data
Federated Learning (FL) allows a number of agents to participate in training
a global machine learning model without disclosing locally stored data.
Compared to traditional distributed learning, the heterogeneity (non-IID) of
the agents slows down the convergence in FL. Furthermore, many datasets, being
too noisy or too small, are easily overfitted by complex models, such as deep
neural networks. Here, we consider the problem of using FL regression on noisy,
hierarchical and tabular datasets in which user distributions are significantly
different. Inspired by Latent Class Regression (LCR), we propose a novel
probabilistic model, Hierarchical Latent Class Regression (HLCR), and its
extension to Federated Learning, FEDHLCR. FEDHLCR consists of a mixture of
linear regression models, allowing better accuracy than simple linear
regression, while at the same time maintaining its analytical properties and
avoiding overfitting. Our inference algorithm, being derived from Bayesian
theory, provides strong convergence guarantees and good robustness to
overfitting. Experimental results show that FEDHLCR offers fast convergence
even in non-IID datasets.