  In many applications, the training data for a machine learning task is partitioned across multiple nodes, and aggregating this data may be infeasible due to storage, communication, or privacy constraints. In this work, we present Good-Enough Model Spaces (GEMS), a novel framework for learning a global satisficing (i.e. "good-enough") model within a few communication rounds by carefully combining the space of local nodes' satisficing models. In experiments on benchmark and medical datasets, our approach outperforms other baseline aggregation techniques such as ensembling or model averaging, and performs comparably to the ideal non-distributed models.
 There has been significant work in designing distributed optimization methods in response to challenges arising from a wide range of large-scale learning applications. These methods typically aim to train a global model by performing numerous communication rounds between distributed nodes. However, most approaches treat communication reduction as an objective, not a constraint, and seek to minimize the number of communication rounds while maintaining model performance. Less explored is the inverse setting-where our communication budget is fixed and we aim to maximize accuracy while restricting communication to only a few rounds. These few-shot model aggregation methods are ideal when any of the following conditions holds:• Limited network infrastructure: Distributed optimization methods typically require a connected network to support the collection of numerous learning updates. Such a network can be difficult to set up and maintain, especially in settings where devices may represent different organizational entities (e.g., a network of different hospitals).• Privacy and data ephemerality: Privacy policies or regulations like GDPR may require nodes to periodically delete the raw local data. Few-shot methods enable learning an aggregate model in ephemeral settings, where a node may lose access to its raw data. Additionally , as fewer messages are sent between nodes, these methods have the potential to offer increased privacy benefits.• Extreme asynchronicity : Even in settings where privacy is not a concern, messages from distributed nodes may be unevenly spaced and sporadically communicated over days, weeks, or even months (e.g., in the case of remote sensor networks or satellites). Few-shot methods drastically limit communication and thus reduce the wall-clock time required to learn an aggregate model.Throughout this paper, we reference a simple motivating example. Consider two hospitals, A and B, which each maintain private (unshareable) patient data pertinent to some disease. As A and B are geographically distant, the patients they serve sometimes exhibit different symptoms. Without sharing the raw training data, A and B would like to jointly learn a single model capable of generalizing to a wide range of patients. The prevalent learning paradigm in this settingdistributed or federated optimization-dictates that A and B share iterative model updates (e.g., gradient information) over a network.From a meta-learning or multitask perspective, we can view each hospital (node) as a separate learning task, where our goal is to learn a single aggregate model which performs well on each task. However, these schemes often make similar assumptions on aggregating data and learning updates from different tasks.As a promising alternative, we present good-enough model spaces (GEMS), a framework for learning an aggregate model over distributed nodes within a small number of communication rounds. Intuitively, the key idea in GEMS is to take advantage of the fact that many possible hypotheses may yield 'good enough' performance for a learning task on local data, and that considering the intersection between these sets can allow us to compute a global model quickly and easily. Our proposed approach has several advantages. First, it is simple and interpretable in that each node only communicates its locally optimal model and a small amount of metadata corresponding to local performance. Second, each node's message scales linearly in the local model size. Finally, GEMS is modular, allowing the operator to tradeoff the aggregate model's size against its performance via a hyperparameter .We make the following contributions in this work . First, we present a general formulation of the GEMS framework. Second, we offer a method for calculating the good-enough space on each node as a R d ball. We empirically validate GEMS on both standard benchmarks (MNIST and CIFAR-10) as well as a domain-specific health dataset. We consider learning convex classifiers and neural networks in standard distributed setups as well as scenarios in which some small global held-out data may be used for fine-tuning. We find that on average, GEMS increases the accuracy of local baselines by 10.1 points and comes within 43% of the (unachievable) global ideal. With fine-tuning, GEMS increases the accuracy of local baselines by 41.3 points and comes within 86% of the global ideal. In summary, we introduce GEMS, a framework for learning an aggregated model across different nodes within a few rounds of communication. We validate one approach for constructing good-enough model spaces (as R d balls) on three datasets for both convex classifiers and simple feedforward networks. Despite the simplicity of the proposed approach, we find that it outperforms a wide range of baselines for effective model aggregation TAB0
We present Good-Enough Model Spaces (GEMS), a framework for learning an aggregate model over distributed nodes within a small number of communication rounds.
{"src": "scitldr-AIC_train", "paper_id": "rJlrN9Bjh4"}