Federated learning improves data privacy and efficiency in machine learning performed over networks of distributed devices, such as mobile phones, IoT and wearable devices, etc. Yet models trained with federated learning can still fail to generalize to new devices due to the problem of domain shift. Domain shift occurs when the labeled data collected by source nodes statistically differs from the target node's unlabeled data. In this work, we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node. Our approach extends adversarial adaptation techniques to the constraints of the federated setting. In addition, we devise a dynamic attention mechanism and leverage feature disentanglement to enhance knowledge transfer. Empirically, we perform extensive experiments on several image and text classification tasks and show promising results under unsupervised federated domain adaptation setting. Data generated by networks of mobile and IoT devices poses unique challenges for training machine learning models. Due to the growing storage/computational power of these devices and concerns about data privacy, it is increasingly attractive to keep data and computation locally on the device (Smith et al., 2017) . Federated Learning (FL) (Mohassel & Rindal, 2018; Bonawitz et al., 2017; Mohassel & Zhang, 2017 ) provides a privacy-preserving mechanism to leverage such decen-tralized data and computation resources to train machine learning models. The main idea behind federated learning is to have each node learn on its own local data and not share either the data or the model parameters. While federated learning promises better privacy and efficiency, existing methods ignore the fact that the data on each node are collected in a non-i.i.d manner, leading to domain shift between nodes (Quionero-Candela et al., 2009) . For example, one device may take photos mostly indoors, while another mostly outdoors. In this paper, we address the problem of transferring knowledge from the decentralized nodes to a new node with a different data domain, without requiring any additional supervision from the user. We define this novel problem Unsupervised Federated Domain Adaptation (UFDA), as illustrated in Figure 1 (a). There is a large body of existing work on unsupervised domain adaptation (Long et al., 2015; Ganin & Lempitsky, 2015; Tzeng et al., 2017; Gong et al., 2012; Long et al., 2018) , but the federated setting presents several additional challenges. First, the data are stored locally and cannot be shared, which hampers mainstream domain adaptation methods as they need to access both the labeled source and unlabeled target data (Tzeng et al., 2014; Long et al., 2017; Ghifary et al., 2016; Sun & Saenko, 2016; Ganin & Lempitsky, 2015; Tzeng et al., 2017) . Second, the model parameters are trained separately for each node and converge at different speeds, while also offering different contributions to the target node depending on how close the two domains are. Finally, the knowledge learned from source nodes is highly entangled (Bengio et al., 2013) , which can possibly lead to negative transfer (Pan & Yang, 2010) . In this paper, we propose a solution to the above problems called Federated Adversarial Domain Adaptation (FADA) which aims to tackle domain shift in a federated learning system through adversarial techniques. Our approach preserves data privacy by training one model per source node and updating the target model with the aggregation of source gradients, but does so in a way that reduces domain shift. First, we analyze the federated domain adaptation problem from a theoretical perspective and provide a generalization bound. Inspired by our theoretical results, we propose an Figure 1 : (a) We propose an approach for the UFDA setting, where data are not shareable between different domains. In our approach, models are trained separately on each source domain and their gradients are aggregated with dynamic attention mechanism to update the target model. (b) Our FADA model learns to extract domain-invariant features using adversarial domain alignment (red lines) and a feature disentangler (blue lines). efficient adaptation algorithm based on adversarial adaptation and representation disentanglement applied to the federated setting. We also devise a dynamic attention model to cope with the varying convergence rates in the federated learning system. We conduct extensive experiments on real-world datasets, including image recognition and natural language tasks. Compared to baseline methods, we improve adaptation performance on all tasks, demonstrating the effectiveness of our devised model. In this paper, we first proposed a novel unsupervised federated domain adaptation (UFDA) problem and derived a theoretical generalization bound for UFDA. Inspired by the theoretical results, we proposed a novel model called Federated Adversarial Domain Adaptation (FADA) to transfer the knowledge learned from distributed source domains to an unlabeled target domain with a novel dynamic attention schema. Empirically, we showed that feature disentanglement boosts the performance of FADA in UFDA tasks. An extensive empirical evaluation on UFDA vision and linguistic benchmarks demonstrated the efficacy of FADA against several domain adaptation baselines.
we present a principled approach to the problem of federated domain adaptation, which aims to align the representations learned among the different nodes with the data distribution of the target node.
{"src": "scitldr-AIC_train", "paper_id": "HJezF3VYPB"}