Federated Learning Hyper-Parameter Tuning from a System Perspective

Federated learning (FL) is a distributed model training paradigm that preserves clients' data privacy. It has gained tremendous attention from both academia and industry. FL hyper-parameters (e.g., the number of selected clients and the number of training passes) significantly affect the training overhead in terms of computation time, transmission time, computation load, and transmission load. However, the current practice of manually selecting FL hyper-parameters imposes a heavy burden on FL practitioners because applications have different training preferences. In this paper, we propose FedTune, an automatic FL hyper-parameter tuning algorithm tailored to applications' diverse system requirements in FL training. FedTune iteratively adjusts FL hyper-parameters during FL training and can be easily integrated into existing FL systems. Through extensive evaluations of FedTune for diverse applications and FL aggregation algorithms, we show that FedTune is lightweight and effective, achieving 8.48%-26.75% system overhead reduction compared to using fixed FL hyper-parameters. This paper assists FL practitioners in designing high-performance FL training solutions. The source code of FedTune is available at https://github.com/DataSysTech/FedTune.


INTRODUCTION
F EDERATED Learning (FL) has been applied to a wide range of applications such as mobile keyboard [1], speech recognition [2], and human stroke prevention [3] on top of mobile devices and Internet of Things (IoT) [4], [5]. Compared to other model training paradigms (e.g., centralized machine learning [6], conventional distributed machine learning [7]), FL has unique properties with regards to (1) Massively Distributed: the number of clients is much larger than the clients' average number of data points; (2) Unbalanced Data: clients have a different amount of data points; and (3) Non-IID Data: each client's data may not represent the overall distribution [8]. In addition to the common hyper-parameters of model training such as learning rates, optimizers, and mini-batch sizes, FL has unique hyper-parameters, including aggregation methods and participant selection [9], [10]. Nonetheless, many FL algorithms, e.g., FedAvg [8], have been proved to converge to the global optimum even different FL hyper-parameters are adopted [11] [12].
Although FL hyper-parameters do not affect FL convergence (i.e., the same final global model accuracy), they significantly affect the training overheads of reaching the final model. Specifically, Computation Time (CompT), Transmission Time (TransT), Computation Load (CompL), and Pengfei Hu is the corresponding author. Transmission Load (TransL) are the four most crucial system overhead: • Computation Time (CompT). It measures how long an FL system spends in model training. Overall model training time must be short for applications that require fast adaptation to environments (e.g., security problems).
• Transmission Time (TransT). It represents how long an FL system spends in model parameter transmission between clients and servers. Without a high-speed transmission solution, transmission time can overwhelm the overall FL training since models are generally large and multiple rounds of model parameter transmissions are required.
• Computation Load (CompL). It is the number of Floating-Point Operation (FLOP) that an FL system consumes. For low-profile devices such as IoT nodes, the computation load must be small as these devices arXiv:2211.13656v1 [cs.LG] 24 Nov 2022 are equipped with low computational resources.
• Transmission Load (TransL). It is the total data size transmitted between the clients and the server. This is critical when data transmission is expensive or its power consumption is a concern. For example, outdoor applications usually rely on cellular communications, which need to pay a considerable price for transmitting a large amount of data. Fig. 1 summarizes the different application scenarios faced by FL practitioners in determining FL hyperparameters. Application scenarios have different training preferences in terms of CompT, TransT, CompL, and TransL. Consider the following examples: (1) attack and anomaly detection in computer networks [13] is time-sensitive (CompT and TransT) as it needs to adapt to malicious traffic rapidly; (2) smart home control systems for indoor environment automation [14], e.g., heating, ventilation, and air conditioning (HVAC), are sensitive to computation (CompT and CompL) because sensor devices are limited in computation capabilities; (3) traffic monitoring systems for vehicles [15] are communication-sensitive (TransT and TransL) because cellular communications are usually adopted to provide city-scale connectivity; (4) precision agriculture based on IoT sensing [16] is not time-urgent but requires energyefficient solutions (CompL and TransL); (5) healthcare systems, e.g., fall detection for elderly people [17], require both fast response and small energy consumption (CompT, TransT, CompL, and TransL); and (6) human stampedes detection/prevention [18] require time, computation, and communication efficient systems.
A few approaches have studied FL training performance under different hyper-parameters [19]. However, they do not consider CompT, TransT, CompL, and TransL together, which is essential from a system's perspective. Besides, they almost all use the number of training rounds for comparison, i.e., round-to-accuracy [12], [20] and convergence rate [8], [11]. The corresponding observations cannot be directly applied to guide FL system design in our context. This is because clients in FL are heterogeneous in terms of their different amounts of local training data (i.e., unbalanced data), computation speeds, and transmission rates, and thus each training round has a different time length, computation cost, and transmission cost. For example, selecting more clients in each training round results in a better round-toaccuracy performance [8], [11]. But it does not necessarily mean a better time-efficiency as the time length of each training round increases with the number of selected clients, nor a better transmission-efficiency as more clients need to transmit model parameters in each training round. In addition, it is challenging to tune multiple hyper-parameters in order to achieve diverse training preferences, especially when we need to optimize multiple system aspects. For example, it is unclear how to select hyper-parameters to build an FL training solution that is both CompT and TransL-efficient. This paper targets a new research problem of optimizing the hyper-parameters for FL from a system perspective. To do so, we formulate the system overheads of FL training and conduct extensive measurements to understand FL training performance. Our measurement results illustrate how to Work Description Single trial System FTS [21] optimize client models DP-FTS-DE [22] trade-off privacy and utility Auto-FedRL [23] improve model accuracy [24] improve training robustness FedEx [25] NAS based framework FLoRA [26] NAS based framework FedTune (Ours) a lightweight framework This paper is organized as follows. First, we introduce related work in Section 2. Then, we clarify the problem and provide intuitive explanations of our preliminary measurement results in Section 3. The details of FedTune algorithm is given in Section 4. Last, we discuss the limitations/opportunities in Section 6 and conclude this paper in Section 7.

RELATED WORK
Hyper-Parameter Optimization (HPO) is a field that has been extensively studied [27]. Many classical HPO algorithms, e.g., Bayesian Optimization (BO) [28], successive halving [29], and hyperband [30], are designed to optimize hyper-parameters of machine learning models. However, they cannot be directly applied to FL because FL has unique hyper-parameters such as the client-side and server-side aggregation methods, and FL is a different training paradigm from centralized machine learning and conventional distributed machine learning. Designing HPO methods for FL is a new research area. Only a few approaches have touched FL HPO problems. Table 1 lists several representative approaches, where we highlight whether the work can execute in an online and single trial manner and whether the work targets system overhead in FL training. For example, BO has been integrated with FL to improve different client models [21] and strength client privacy [22]. Several approaches apply reinforcement learning to adjust FL hyper-parameters [23], [24], which introduces extra complexity and loss of generality. FedEx is a general framework to optimize the round-to-accuracy of FL by exploiting the Neural Architecture Search (NAS) techniques of weight-sharing, which improves the baseline by several percentage points [25]; FLoRA determines the global hyper-parameters by selecting the hyper-parameters that have good performances in local clients [26]. Recently, a benchmark suite for federated hyper-parameter optimization [31] is designed, whose effectiveness remains to be investigated.
For two reasons, existing approaches cannot be directly applied to our scenario of optimizing FL hyper-parameter for different FL training preferences. First, CompT (in seconds), TransL (in seconds), CompL (in FLOPs), and TransL (in bytes) are not comparable with each other. Incorporating training preferences in HPO is not trivial. Second, hyperparameter tuning needs to be done during the FL training. No "comeback" is allowed as the FL model keeps training until its final model accuracy. Otherwise, it will cause significantly more system overhead.

UNDERSTANDING THE PROBLEM
We first quantify the system overhead of FedAvg to illustrate the problem. FedAvg minimizes the following objective (1) where f i (w) is the loss of the model on data point (x i , y i ), that is, f i (w) = (x i , y i ; w), K is the total number of clients, P k is the set of indexes of data points on client k, with n k = |P k |, and n is the total number of data points from all clients, i.e., n = K k=1 n k . Due to the large number of clients in a typical FL application (e.g., millions of clients in the Google Gboard project [1]), a common practice is to  randomly select a small fraction of clients in each training round. In the rest of this paper, we refer to the selected clients as participants and denote M as the number of participants in each training round. Each participant makes E training passes over its local data in each round before uploading its model parameters to the server for aggregation. Afterward, participants wait to receive an updated global model from the server, and a new training round starts.

System Model
Assume that clients are homogeneous regarding hardware (e.g., CPU/GPU) and network (e.g., transmission speeds). Let b k,r indicate whether client k participates at the training round r. Then, we have K k=1 b k,r = M , i.e., each round selects M participants. The number of training rounds to reach the final model accuracy is denoted by R, which is unknown a priori and varies when different sets of FL hyper-parameters are used in FL training. CompT, TransT, CompL, and TransL can be formulated as follows.
• Computation Time (CompT). If client k is selected in a training round, its training time can be represented by C 1 ·E·n k , where C 1 is a constant. It is proportional to its number of data points (i.e., n k ) because n k decides the number of local updates (number of mini-batches) for one epoch, and each local update includes one forward-pass and one backward-pass. The computation time of the training round r is determined by the slowest participant and thus is represented by C 1 · E · max K k=1 b k,r · n k . In total, the computation time of an FL model training can be formulated as • Transmission Time (TransT). Each participant in a training round needs one download and one upload of model parameters from and to the server [19]. Thus, the transmission time is the same for all participants in any training round, i.e., a constant C 2 . The total transmission time is represented by  • Computation Load (CompL). Client k has C 3 · E · n k computation load if it is selected in a training round, where C 3 is a constant. The computation load of the training round r is the summation of each participant's computation load and thus is C 3 ·E· K k=1 b k,r · n k . We can formulate the overall computation load as • Transmission Load (TransL). Since each training round selects M participants, the transmission load for a training round is C 4 · M where C 4 is a constant. The total number of training rounds is R, and thus, the total transmission load of an FL training is represented by As we assume that clients have homogeneous hardware and network, clients have the same C 1 , C 2 , C 3 , and C 4 . Thus, these constants do not affect the comparison of training overhead under different hyper-parameters when the same model is used. In the experiments, we assign the model's number of FLOPs for one input to C 1 and C 3 , and the model's number of parameters to C 2 and C 4 .

Measurement Setup
We conduct measurements to study the system overheads when different FL hyper-parameters are used for training. The measurements aim to help us better understand FL training and are the basis of our automatic tuning algorithm. We use the Google speech-to-command dataset [32], which classifies one second's audio clip into 35 commands such as yes, no, right, and up. The dataset includes audio clips that are crowd-sourced from 2618 clients. As officially suggested in [32], we use 2112 clients' data for training and the remaining 506 clients' data for testing. Fig. 2(a) shows the distribution of the number of clients versus the number of data points. It is clear that clients' data are heterogeneous: many clients have only one data point, while others can have up to 316 data points. Fig. 2(b) plots the histogram of each class's number of data points, which shows that the overall data distribution is unbalanced. The speech-tocommand dataset demonstrates the three data properties of FL: massively distributed, unbalanced, and non-IID. In the measurement study, we investigate the FL training overhead in terms of the following three hyper-parameters.

•
The number of participants (i.e., M ). It is well-known that more participants in each training round have a better round-to-accuracy performance [8]. However, the time efficiency is not necessarily better because the time length of each training round also increases with more participants. Besides, it is unclear how computation and communication overheads behaves for different number of participants. In the measurement study, we set M to 1, 10, 20, and 50.

•
The number of training passes (i.e., E). Increasing the number of training passes as a method to improve communication efficiency has been adopted in several works, such as FedAvg [8] and FedNova [11]. However, how does the number of training passes affect the time overhead and computation overhead is unclear. In the measurement study, we set E to 0.5, 1, 2, 4, 8, where 0.5 means that only half of each client's local data are used for local training in each round.
• Model complexity. We also investigate how the model complexity influences the training overheads if a target accuracy is met. Although it is common knowledge that smaller models have better time and computation performance in other paradigms of model training, we are the first to report the four system overhead versus model complexity in the FL setting. We use ResNet [33] to build different models, as listed in Table 2.

Illustration of FL Training
To gain an intuitive understanding of FL training, we plot FL training profiles in Fig. 3 where a different number of participants M is used on each round. We use ResNet-18 and set the target model accuracy to 0.8. In this measurement, E, C 1 , C 2 , C 3 and C 4 are all set to 1, for illustration purpose. Results are normalized to the largest overhead. As expected, more participants lead to a better accuracyto-round performance ( Fig. 3(a)). However, if the FL training is stopped too early, it is misleading to conclude that more participants result in higher final model accuracy. Instead, they all reach the same model accuracy as FL hyperparameters do not invalidate the convergence. Correspondingly, more participants have a better accuracy-to-CompT performance, as shown in Fig. 3(b). However, the performance gap between a different number of participants for the accuracy-to-CompT is less significant than the accuracyto-round performance. This is because the time duration of each training round increases when more participants are selected on each training round (Fig. 3(c)). In this measurement, there are no obvious accuracy-to-CompT difference when 20 and 50 participants are selected. On the other hand, fewer participants lead to better accuracy-to-CompL performance, as shown in Fig. 3(d), which means the effectiveness of each unit computation operation is higher when fewer participants are involved in the FL training. The accuracyto-TransT ( Fig. 3(e)) is the same as the accuracy-to-round ( Fig. 3(a)) since we adopt C 3 = 1 for plotting. That is, more participants have a better accuracy-to-TransT performance. Fig. 3(f) shows that the accuracy-to-TransL performance has the opposite trend, i.e., more participants result in a worse performance.

Measurement Results
In Section 3.3, we illustrate CompT, CompL, TransT, and TransL performance when a different number of partic- Training aspect M E Model complexity ipants are adopted in FL training. This section presents results from more extensive measurements for the number of participants M , the number of training passes E, and the model complexity. The results are averaged over three runs. In the experiments, we use ResNet-18 and normalize their overheads. As we can see, more participants lead to smaller CompT, i.e.; it takes a shorter time to converge. However, the difference is insignificant among 10, 20, and 50 participants, especially when the number of training passes is large. In addition, we can see that larger E has worse CompT. There is no apparent difference between E = 0.5 and E = 1 though. In a nutshell, the common knowledge of more participants is faster for FL model training is valid. However, the gain of more participants is insignificant when the number of participants is moderate. In addition, it is preferred to adopt a small number of training passes to achieve good time efficiency.
• Transmission Time (TransT). Fig. 4(b) plots TransT, which clearly shows that TransT favors larger M and E. Since TransT is dependent on the number of training rounds R (Eq. (3)), it is equivalent to the metric of round-to-accuracy. Our measurement result is consistent with common knowledge (e.g., [12]) that more participants and more training passes have a better round-to-accuracy performance. We can also observe that when M is small, e.g., 1, TransL is much worse than the other cases.
• Computation Load (CompL). Fig. 4(c) shows CompL. We make the following observations: (1) More participants result in worse CompL. The results indicate that the gain of faster model convergence from more participants does not compensate for the higher computation costs introduced by more participants. (2) CompL is increased when more training passes are used. This is probably because that larger E diverges the model training [20], and thus, the data utility per unit of computation cost is reduced.
• Transmission Load (TransL). Fig. 4 Table 2 tabulates the models for comparing training overheads versus model complexity. In this experiment, we select one participant (M = 1) to train one pass (E = 1) on each training round. Fig. 5 shows the normalized CompT, TransT, CompL, and TransL for different models. The xaxis is the target model accuracy, and the y-axis is the corresponding overhead to reach that model accuracy. Since only one client and one training pass are used on each round, CompT and CompL have the same normalized comparison, and so are TransT and TransL. The results show that smaller models are better in terms of all training aspects. In addition, it is interesting to note that heavier models have higher increase rates of overheads versus model accuracy. This means that model selection is especially essential for high model accuracy applications.

Summary of System Overhead
Based on our measurement study, we summarize systems overheads versus FL hyper-parameters in Table 3. As we can see, CompT, TransT, CompL, and TransL conflict with each other in selecting the optimal M and E. Regarding model complexity, smaller models have better system overheads if the model accuracy is satisfied. Please note that Table  3 is consistent with existing work (e.g., [12]), but is more comprehensive. Thus, Table 3 is also valid for other datasets and ML models.

Intuitive Explanation of Measurement Results
We present intuitive explanation of the measurement results to help FL practitioners better understand FL hyperparameters. Fig. 6 visualizes the intuition.

•
The number of participants M . In Fig. 6(a), the top scheme and the bottom scheme have the same computation cost (two local training) and communication cost (four transmissions). However, the bottom scheme has a better overall CompL and TransL at the expense of degraded CompT and TransT. The bottom scheme is better probably because the clients in the bottom scheme always work on the updated global model, whereas the clients in the top scheme work on the same global model. In other words, narrowand-deep FL schemes have better computation load and transmission load than wide-and-shallow FL schemes.
• The number of training passes E. In Fig. 6(b), both the top scheme and the bottom scheme take E total training passes. However, the bottom scheme has better CompT and CompL at the expense of higher communication costs. The results indicate that the usefulness per local update decreases with the number of local updates. Therefore, for computationsensitive FL applications, large E should be avoided.
• Model complexity. Fig. 6(c) shows that a smaller model has better CompT, TransT, CompL, and TransL than a heavier model, as long as the accuracy requirement is met. In addition, our results in Fig. 5 indicate that if the target model accuracy is low, then using a heavy model does not introduce significantly more overhead than a lightweight model. However, if the goal is to achieve a high-accuracy model, carefully selecting a model complexity is essential, as overlarge models cause significantly more system overhead.

FEDTUNE: AUTOMATIC TUNING OF FEDERATED LEARNING HYPER-PARAMETERS FROM A SYSTEM PERSPECTIVE
Due to the conflicting FL hyper-parameters for CompT, TransT, CompL, and TransL, practitioners bear the burden of selecting a set of FL hyper-parameters, which if not chosen well may lead to poor system performance. For example, it is unclear how to set FL hyper-parameters that are both CompT and TransT friendly, since these two system aspects prefer different number of training passes. We propose FedTune to adjust FL hyper-parameters during the FL training automatically. FedTune considers the application's preference for CompT, TransT, CompL, and TransL, denoted by α, β, γ, and δ, respectively. We have α+β +γ +δ = 1. For example, α = 0.6, β = 0.2, γ = 0.1, and δ = 0.1 represent that the application cares most about CompT, somewhat about TransT, and the least on CompL and TransL.

Problem Formulation
For two sets of FL hyper-parameters S 1 and S 2 , FedTune defines the comparison function I(S 1 , S 2 ) as where t 1 and t 2 are CompT for S 1 and S 2 when achieving the same model accuracy. Correspondingly, q 1 and q 2 are TransT, z 1 and z 2 are CompL, and v 1 and v 2 are TransL. If I(S 1 , S 2 ) < 0, then S 2 is better than S 1 . A set of hyper-parameters is better than another set if the weighted improvement of some training aspects (e.g., CompT and CompL) is higher than the weighted degradation, if any, of the remaining training aspects (e.g., TransT and TransL). The weights are training preferences on CompT, TransT, CompL, and TransL. However, the training overhead for different sets of FL hyper-parameters are unknown a priori. As a result, directly identifying the optimal hyper-parameters before FL training is impossible. Instead, we propose an iterative method to optimize the next set of hyper-parameters. Given the current set of hyper-parameters S cur , the goal is to find a set of hyper-parameters S nxt that improves the training performance the most, that is, minimizes the following objective function: where t cur , q cur , z cur , and v cur are CompT, TransT, CompL, and TransL under the current hyper-parameters S cur ; t nxt , q nxt , z nxt , and v nxt are CompT, TransT, CompL, and TransL for the next hyper-parameters S nxt . We focus on the number of participants M and the number of training passes E, since training overhead is monotonous with model complexity. Therefore, we need to optimize S nxt = {M nxt , E nxt }.

S nxt Optimization
To find the optimal S nxt , we take the derivatives of G(S nxt ) over M and E, obtaining Algorithm 1: FedTune update.  We illustrate how to approximate ∆M . The process for ∆E is similar. Considering that each step makes a small adjustment of M , ∂t nxt /∂M can be represented by (+1)×|t nxt −t cur |, where (+1) means CompT prefers larger M according to Table 3. To estimate |t nxt − t cur |, we apply a linear function η t−1 ×|t cur −t prv | where η t−1 = |tcur−tprv| |tprv−tprvprv| (t prvprv is the CompT at two steps before). Thus, η t−1 represents the slope of the linear function. Similarly, we have η q−1 , η z−1 , η v−1 for TransT, CompL, and TransL when calculating their derivatives over M . As a result, ∆M can  be approximated as Similarly, we can calculate ∆E as where ζ t−1 , ζ q−1 , ζ z−1 , and ζ v−1 are the parameters for calculating the derivatives of CompT, TransT, CompL, and TransL over E. FedTune automatically updates η t−1 , η q−1 , η z−1 , η v−1 , ζ t−1 , ζ q−1 , ζ z−1 , and ζ v−1 during FL training. At each step, FedTune updates the parameters that favor the current decision. For example, if M cur is larger than M prv , FedTune updates η t−1 and η q−1 as CompT and TransT prefer larger M (line 17); otherwise, FedTune updates η z−1 and η v−1 .

Decision Making and Parameter Update
Furthermore, FedTune incorporates a penalty mechanism to mitigate bad decisions. Given the previous hyperparameters S prv and the current hyper-parameters S cur , FedTune calculates the comparison function I(S prv , S cur ) (line 15). A bad decision occurs if the sign of I(S prv , S cur ) is positive (line 18). In this case, FedTune multiplies the parameters that are against the current decision by a constant penalty factor, denoted by D (D ≥ 1). For example, if I(S prv , S cur ) > 0 and M cur > M prv , FedTune updates η t−1 and η q−1 as explained before, but also multiplies η z−1 and η v−1 by D (line 19-20).

EVALUATION
In this section, we provide the evaluation results of FedTune. First, we explain our experiment setup in Section 5.1. Next, we present the overall performance in Section 5.2. Then, we conduct trace analysis in Section 5.3. Last, the penalty mechanism is studied in Section 5.4.

Experiment Setup
Benchmarks and Baseline. We evaluate FedTune on three datasets: speech-to-command [32], EMNIST [34], and Cifar-100 [35], and three aggregation methods: FedAvg [8], Fed-Nova [12], and FedAdagrad [36]. We set equal values for the combination of training preferences α, β, γ and δ (see the first column in Table 4). Therefore, for each dataset, we conduct 15 combinations of training preferences. We set target model accuracy for each dataset and measure CompT, TransT, CompL, and TransL for reaching the target model accuracy. We regard the practice of using fixed M and E as the baseline and compare FedTune to the baseline by calculating Eq. (6). In the evaluation, the positive performance means FedTune reduces the system overheads and the negative performance means the degradation. We implemented FedTune in PyTorch. All the experiments are conducted in a server with 24-GB Nvidia RTX A5000 GPUs.

Training Setup.
(1) speech-to-command dataset. It classifies audio clips to 35 commands (e.g., 'yes', 'off'). We transform audio clips to 64-by-64 spectrograms and then downsize them to 32-by-32 gray-scale images. As officially suggested [32], we use 2112 clients' data for training and the remaining 506 clients' data for testing. We set the minibatch size to 5, considering that many clients have few data points. We use ResNet-10 and the target model accuracy of 0.8. (2) EMNIST dataset. It classifies handwriting (28-by-28 gray-scale images) into 62 digits and letters (lowercase and uppercase). We split the dataset based on the writer ID. We randomly select 70% writers' data for training and the remaining for testing. We use a Multiplayer Perception (MLP) model with one hidden layer (200 neurons with ReLu activation). We set the mini-batch size to 10 and the target model accuracy of 0.7. (3) Cifar-100 dataset. It classifies 32by-32 RGB images to 100 classes. We randomly split the dataset into 1200 users, where each user has 50 data points. Then, we randomly select 1000 users for training and the remaining 200 users for testing. We set the mini-batch size to 10. ResNet-18 is used, and the target model accuracy is set to 0.2 (due to our limited computational capability, we set a low threshold for Cifar-100).
For all datasets, we normalize the input images with the mean and the standard deviation of the training data before feeding them to models for training and testing. Both M and E are initially set to 20. FedTune is activated when the model accuracy is increased by at least 0.01 (i.e., = 0.01). The penalty factor D is set to 10

Overall Performance
Results for Diverse Datasets. Table 5 shows the overall performance of FedTune for different datasets when FedAvg is applied. We set the learning rate to 0.01 for the speech-tocommand dataset and the EMNIST dataset, and 0.1 for the Cifar-100 dataset, all with the momentum of 0.9. We show the standard deviation in parenthesis. As shown, FedTune consistently improves the system performance across all the three datasets. In particular, FedTune reduces 22.48% system overhead of the speech-to-command dataset compared to the baseline by averaging the 15 combinations of training preferences. We also observe that the FL training benefits more from FedTune if the training process needs more training rounds to converge. Our experiments with EMNIST (small model) and Cifar100 (low target accuracy) only require a few dozens of training rounds to reach their target model accuracy, and thus their performance gains from FedTune are not significant. The observation is consistent with the decision-making process in FedTune, which increases/decreases hyper-parameters by only one at each step. We leave it as future work to augment FedTune to change hyper-parameters with adaptive degrees.
Results for Different Aggregation Methods. Table 6 shows the overall performance of FedTune for different aggregation methods when we use the speech-to-command dataset and the ResNet-10 model. We set the learning rate to 0.1, β 1 to 0, and τ to 1e-3 in FedAdagrad. As shown, FedTune achieves consistent performance gain for diverse aggregation methods. In particular, FedTune reduces the system overhead of FedAdagrad by 26.75%.

Trace Analysis
We present the details of traces when the speech-tocommand dataset and the FedAdagrad aggregation method are used. Table 4 tabulates the results, where we show the application preference (α, β, γ, and δ), the system overheads (CompT, TransT, CompL, and TransL), the final M and E when the training is finished, and the overall performance. We report the average performance, as well as their standard deviations in parentheses. The first row is the baseline, which does not change hyper-parameters during the FL training. As we can see from Table 4, FedTune can adapt to different training preferences. Specifically, FedTune reduces the system overhead up to 70.51% when the application only cares about computation load (i.e., γ = 1). Only one preference (0.33, 0.33, 0, 0.33) results in a slightly degraded performance. On average, FedTune improves the overall performance by 26.75% for the speech-to-command dataset and the FedAdagrad aggregation method. Fig. 7 illustrates one trace of M and E during the FL training for each application preference. The experiment is conducted with the speech-to-command dataset and the FedAdagrad aggregation method. The experiment result clearly shows that FedTune can automatically adjust FL hyper-parameters during the FL training while respecting the application's preference on system overhead. We also observe that FedTune does not keep increasing or decreasing M and E. Instead, in many cases, it intelligently tunes the M and E during the different phases of the FL training.

Study of Penalty Mechanism
We investigate our penalty mechanism by conducting experiments with the Google speech-to-command dataset and Fig. 9. Performance comparison of FedTune without penalty mechanism and with penalty mechanism for the Speech-to-command dataset and FedAvg aggregation method.
First, we conduct experiments to explore whether the penalty mechanism can mitigate the degradation. Results are averaged over three runs. Fig. 8 plots the performance of FedTune for the degraded cases versus penalty factors, where error bars represent the standard deviation. Although the penalty mechanism does not guarantee positive performance for the degraded cases, it mitigates the degradation. Fig. 8 also shows that FedTune remains stable for a moderate value of penalty factors. Empirically, we set the penalty factor to 10 in FedTune. Fig. 9 compares the performance of FedTune when the penalty factor is 10 (full-fledged) versus 1 (no penalty) for different training preferences. Results are averaged by three runs and the error bars represent the standard deviation. Overall, FedTune increases the performance gain of the FedAvg aggregation method from 17.97% to 22.48%. In addition, FedTune with the penalty mitigates the highly degraded cases and thus is more practical than the nonpenalty counterpart. We also observe that FedTune with penalty mechanism is more stable, with the averaged standard deviation of 7.77% versus 14.14% in the non-penalty counterpart.

DISCUSSION
FedTune has promising performance in tuning FL hyperparameters. As one of the first work of its kind, FedTune has some limitations/opportunities that deserve further exploration.
Heterogeneous Devices. This paper assumes that clients are homogeneous, i.e., the same hardware and network. In practice, however, client devices are heterogeneous. Measurements of the computation capabilities of mobile devices and their network throughput exhibit order-of-magnitude difference [37], [38], [39]. As a result, for clients even with the same amount of local data points, they result in different computation and transmission costs. Currently, FedTune only tunes the system-wide hyper-parameters, which might not be optimal for heterogeneous devices that may require device-level hyper-parameter tuning. However, compared to the overwhelming and laboring effort of manually determining the FL hyper-parameters for each device, tuning system-wide hyper-parameter seems to be a more practical approach for FL practitioners. We leave it as future work to evaluate FedTune on heterogeneous devices.
Extensions. Many FL algorithms have been proposed to tackle the limitations of FedAvg. FedTune could be extended to the following scenarios. (1) Participant selection. Compared to the random selection of participants, guided participant selection that considers clients' data utility and device utility can improve overall training performance [9]. Popular alternatives are to only wait for participants that are finished before a deadline [40] or only wait for the first M participants [11]. (2) Adaptive training passes across participants. Due to the heterogeneity of clients, setting the same number of training passes E for all participants in each training round is not optimal. To support different E across participants, FedNova [12] relies on re-weighting of aggregation while FedProx [20] adds a proximal term to stabilize the convergence. We plan to incorporate these functionality to further improve the performance of Fed-Tune. (3) Investigate more FL hyper-parameters. Although we only take the number of participants, the number of training passes, and the model complexity as examples to illustrate the workflow of FedTune, we believe FedTune is a general framework, which can be applied to other FL hyperparameters, e.g., the mini-batch size.

CONCLUSION
FL involves high system overhead in the training process, which hinders its research and real-world deployment. We argue that optimizing system overhead for FL applications is extremely valuable. To this end, we propose FedTune to adjust FL hyper-parameters, catering to the application's training preferences automatically. Our evaluation results show that FedTune is general, lightweight, flexible, and is able to significantly reduce system overhead in FL training.