AgEBO-Tabular: Joint Neural Architecture and Hyperparameter Search with Autotuned Data-Parallel Training for Tabular Data

Developing high-performing predictive models for large tabular data sets is a challenging task. The state-of-the-art methods are based on expert-developed model ensembles from different supervised learning methods. Recently, automated machine learning (AutoML) is emerging as a promising approach to automate predictive model development. Neural architecture search (NAS) is an AutoML approach that generates and evaluates multiple neural network architectures concurrently and improves the accuracy of the generated models iteratively. A key issue in NAS, particularly for large data sets, is the large computation time required to evaluate each generated architecture. While data-parallel training is a promising approach that can address this issue, its use within NAS is difficult. For different data sets, the data-parallel training settings such as the number of parallel processes, learning rate, and batch size need to be adapted to achieve high accuracy and reduction in training time. To that end, we have developed AgEBO-Tabular, an approach to combine aging evolution (AgE), a parallel NAS method that searches over neural architecture space, and an asynchronous Bayesian optimization method for tuning the hyperparameters of the data-parallel training simultaneously. We demonstrate the efficacy of the proposed method to generate high-performing neural network models for large tabular benchmark data sets. Furthermore, we demonstrate that the automatically discovered neural network models using our method outperform the state-of-the-art AutoML ensemble models in inference speed by two orders of magnitude while reaching similar accuracy values.


INTRODUCTION
Tabular data sets are often diverse. They are obtained from multiple sources and modes, where combining certain inputs using problemspecific domain knowledge typically leads to better and physically meaningful features and consequently robust models [10,39]. Many high-performing predictive models for tabular data are based on classical supervised machine learning (ML) methods such as bagging, boosting, and kernel-based methods. Specifically, ensemble methods that combine models obtained from different supervised ML methods have emerged as state-of-the-art for a wide range of predictive modeling tasks with tabular data. However, the design and development of such ensemble models is a highly iterative, manually intensive, and time-consuming task. Typically an ML pipeline for tabular data is composed of several components: data processing, dimension reduction, data balancing, feature selection, hyperparameter tuning, model selection, and ensemble strategy (such as stacking, bagging, and weighted combination). Given the design choices for each component, the complexity of designing an effective ML pipeline for tabular data is often beyond nonexperts.
Deep neural networks (DNNs) have achieved significant success in overcoming the issues of manual feature engineering and the complexities of developing a classical supervised ML pipeline. Nevertheless, designing DNNs for tabular data has received relatively less attention compared to image and text data. From the methodological perspective, there are two main reasons.
First, given the diversity of tabular data, designing DNNs with shared patterns such as convolutional and recurrent units is not meaningful unless further assumptions about the data are made. Second, fully connected DNNs, which are typically used for tabular data, can potentially lead to unsatisfactory performance because they can have large numbers of parameters, overfitting issues, and a difficult optimization landscape with low-performing local optima [11].

arXiv:2010.16358v2 [cs.LG] 26 Oct 2021
Automated machine learning (AutoML) is a promising approach to address the methodological challenges in developing DNNs for tabular data. Neural architecture search (NAS), a class of AutoML, is an approach to automate the development of customized DNNs for a given data set. The NAS methods can be grouped into individual search methods and weight-sharing methods. The former generate a large number of architectures from a user-defined search space, train and validate each of them, and use the accuracy values to improve the generated architectures. The main advantage of these methods is parallelization: the generated architectures are independent, and they can be trained simultaneously. The disadvantage is that since each architecture is trained from scratch, architecture evaluation is expensive and becomes a bottleneck for effectiveness. To alleviate this issue, researchers proposed a different approach where the trained weights or computations are shared from an architecture to another during the search. This is enabled by defining a search space as an overparameterized network [26] (also named hypernetwork), where the search samples subarchitectures and leverages the trained weights and computations from previously trained subarchitectures. This results in a significant reduction of evaluation time for several tasks. Nevertheless, the disadvantage of these methods is the instability due to the optimization gap between the supernetwork and its subarchitectures. In particular, optimizing the hypernetwork does not necessarily result in high-quality subarchitectures [8].
We focus on individual NAS for large tabular data because of its ability to leverage multiple compute nodes to find high-performing neural networks. Specifically, we adopt aging evolution (AgE) [28], a parallel NAS method that generates a population of neural architectures, trains them concurrently using multiple nodes, and improves the population by performing mutations on the existing architectures within a population. To reduce the training time of each architecture, we utilize the widely used distributed dataparallel training technique.
In this approach, the large training data is split into shards and distributed to multiple processing units. Multiple models with the same architecture are trained on different data shards, and the gradients from each model are averaged and used to update the weights of all models. Combining an individual NAS search method with distributed data-parallel training is a challenging task because the combination of the two methods requires nested parallelism. Moreover, the distributed data parallelism requires data-set-specific tuning of learning rate, and batch size in order to maintain accuracy and reduce training time.
To that end, we make the following contributions: • We develop AgEBO-Tabular, a joint neural architecture and hyperparameter search that combines aging evolution (AgE), a parallel NAS method [28] for searching the neural architecture space, and an asynchronous Bayesian optimization method for tuning the hyperparameters of data-parallel training. AgEBO-Tabular searches the architecture space and the hyperparameters of data-parallel training simultaneously. • We evaluate the efficacy of the proposed approach on two large tabular data sets from ECP-CANDLE benchmarks and show that AgEBO outperforms the accuracy of the AgE and discovers architectures that are faster to train.
• We show that models produced by AgEBO outperform the manually designed models on the two ECP-CANDLE benchmark data sets.
The novelty of our work is fourfold: developing a new method for joint neural architecture and hyperparameter search, accelerating NAS with data-parallel training, using asynchronous Bayesian optimization for tuning the hyperparameters of data-parallel training, and advancing the state-of-the-art in the design of DNNs for large tabular data.

PROBLEM FORMULATION
Let , , and be the training, validation, and test data, respectively. A neural architecture configuration ℎ is a vector from the neural architecture search space , defined by a set of neural architecture decision variables. A hyperparameter configuration ℎ is a vector from hyperparameter search space defined by a set of hyperparameters. The joint neural architecture and hyperparameter search space is given by × . The problem of joint neural architecture and hyperparameter search can be formulated as the following bilevel optimization problem: where M * (ℎ , ℎ ) is the validation accuracy that needs to be maximized on and L ℎ ,ℎ ( ) is a loss function that needs to be minimised by optimizing the weights of the neural network configured with (ℎ ,ℎ ) using . The test data is used only for the final evaluation.
The architecture search space differs from the hyperparameter search space with respect to the values that the decision variables take. All the decision variables in the architecture search space belong to the categorical (nonordinal) type, where different values for a given variable do not have any particular order. On the other hand, the hyperparameter search space is characterized by mixed-integer variables. This comprises integer, real, binary, and categorical types. Often, the number of categorical hyperparameters is relatively smaller than that of other types. Note that when all variables in the hyperparameter search space belong to a categorical type, explicit partitioning in the search space is not required; consequently, a custom method such as our proposed AgEBO-Tabular for joint neural architecture and hyperparameter search becomes less relevant.
In our study, is defined by the decision variables to construct fully connected neural networks with skip connections for tabular data, and is defined by the hyperparameters of the data-parallel training (learning rate, batch size, optimizer, patience for learning rate reduction, patience for early stopping and loss function).

AGEBO-TABULAR
The AgEBO-Tabular approach that we propose comprises three components: neural architecture search space for tabular data, dataparallel training as evaluation strategy, and the AgEBO algorithm for joint neural architecture and hyperparameter search.

Neural architecture search space for tabular data
We model the search space of the neural architecture using a directed acyclic graph, which starts and ends with input and output nodes, respectively. They are fixed based on the input and output dimensions of the tabular data, respectively. It is possible to have multiple input nodes as well as output nodes. Between these, two sets of nodes are intermediate nodes, each of which can be a variable N or a skip-connection SC node. Each node represents a categorical decision variable that can take a list of nominal values (i.e., without order). Each variable node represents a dense layer with a list of different layer types; the choice is made by the NAS method. The skip connections between the variable nodes are created by using skip-connection nodes. This type of node has two choices: zero for no skip connection and identity for the creation of skip connection.
If multiple inputs are defined in the data, then subgraphs will be created for each of these inputs such as shown in Figure 1a. However, inputs with equal shape will benefit of parameter sharing and therefore be processed by the same subgraph.
Given a pair of consecutive variable nodes N , N +1 , three skipconnection nodes SC +1 −3 , SC +1 −2 , SC +1 −1 are created. The choice of identity for these skip-connection nodes respectively allows for connection to the three previous nonconsecutive variable nodes For example, if an identity is chosen for SC +1 −1 , a skip connection is made between N −1 and N +1 by passing the tensor output from N −1 through a merging operator (e.g., concatenation, sum after projection or padding).
In the case of sum, a linear layer is used to project the tensor from N −1 to a correct shape. This is required for the creation of skip connections between N −1 and N +1 when their number of neuron units is different. The sum operator adds the projected input tensor from N −1 and the tensor from N , passes the summed tensor through the activation function, and sends the resulting tensor as input to N +1 . When SC +1 −2 and SC +1 −3 take identity values, the tensors from N −2 and N −3 undergo the same linear projection, and the tensor is given to the sum operator. When there is no skip connection, SC +1 −3 , SC +1 −2 , SC +1 −1 are set to zero; N and N +1 are fully connected without the linear layer and the sum operator. In the case of concatenation, the tensors are passed when there is a connection and simply concatenated together.
The same process is repeated for each of the variable nodes. See Figure 1b for an example.
The dense layer type is defined by the number of units and the activation function. For the former and the latter we used values in (50, 2000) with a step of 25 and a {Identity, Swish [27], ReLu, Tanh, Sigmoid}. These resulted in 391 (78 units × 5 activation functions, and identity) dense layer types for each variable node. Although one can order 391 values using the number of units in the layer, we did not consider and leverage such an order from the generality perspective.
For example, if we consider only one value for the unit and different activation functions, then we cannot order the values in the list and cannot leverage the ordering in the NAS. We set the maximum number of variable nodes to 5 for each sub-graph.
The first variable node will not have a skip connection node. The second and the third variable nodes have 1 and 2 skip-connection nodes, respectively. The fourth to tenth variable nodes have 3 skipconnection nodes each. The output node has 3 skip connections as well.
(a) Global search space.
(b) Detailed sub-graph. Figure 1: Neural architecture search space. The global search space is shown in 1a. The detailed structure of a sub-graph is shown in 1b. The nodes N 1 and N 2 represent dense layers ( , ), where is the number of neurons and is the activation function. The nodes SC 2 1 , SC 3 1 , SC 3 2 represent the possible skip-connection nodes, when R is chosen for each of them. The nodes shown in red with Σ are used to represent the merging operators (sum or concatenation).

Data-parallel training as evaluation strategy
The evaluation of an architecture in the individual NAS method consists of training the network and computing the validation accuracy. To speed up the evaluation, we use distributed data-parallel training. Given a neural architecture A, the training data set is split in mutually exclusive subsets called shards, which are given to parallel processing units. Each of the processing units trains a copy of the same neural architecture A on its own shard. The gradients from each copy of neural architecture are synchronized and are used to update the weights. Moreover, we use the widely used linear scaling rule [16] to adapt the learning rate and batch size depending on the level of parallelism in the data-parallel training. This heuristic states that the learning rate and batch size with processes should be scaled linearly with respect to : where 1 , 1 , are respectively the learning rate and batch size used for training with a single process. We treat , 1 , and 1 as hyperparameters and tune them using Bayesian optimization. By leveraging the linear scaling rule, we try to achieve linear scaling for training time; however, there is an upper linear scaling limit above which the accuracy will suffer (without advanced and sophisticated layer-wise learning rate and adaptive batch size). Therefore, by tuning , 1 , and 1 , we try to find the upper linear scaling limit that gives a maximal reduction in training time without losing accuracy.

AgEBO: Aging evolution with Bayesian optimization
To perform a joint neural architecture and hyperparameter search, we propose aging evolution with Bayesian optimization (AgEBO). Our method combines AgE, a parallel NAS method for searching over the architecture space, and asynchronous Bayesian optimization (BO), for tuning the hyperparameters data-parallel training. There is not a lot of literature dedicated to asynchronous BO [3] compared to batch-synchronous BO [7]. However, the asynchronous approach is justified by our practical case where the evaluated function can have significantly different runtimes depending on the architecture and hyperparameter configuration (see tables 2b and 2a). A way to easily parallelize BO is to use the constant liar heuristic such as described in [15,31].
Algorithm 1 shows the pseudo code of AgEBO. The method follows the manager-worker paradigm for parallelization. It starts with workers, each with a maximum of parallel processing units for data-parallel training. The initialization phase starts by allocating an empty queue for the population of size and BO optimizer object. It is followed by sampling architecture configurations and hyperparameter configurations, respectively, and concatenating them. The neural network models are built by using the resulting configurations and are sent for concurrent evaluation on workers by using the submit_evaluation interface (lines 3-7). Each worker uses the learning rate, batch size, and the number of processes from the configuration that it received to run the dataparallel training. The iterative part of the algorithm consists of collecting the results (validation accuracy values) once the workers finish their evaluation (line 9) and using them for generating the next set of architecture and hyperparameter configurations for evaluation. The BO optimizer object takes the hyperparameter configurations and their corresponding validation accuracy values and generates a | | number of hyperparameter values (using optimizer.tell and optimizer.ask interfaces, respectively, lines [12][13]. To generate | | number of architecture configurations, the following steps are performed repeatedly: random sampling architecture configurations from the incumbent population, selecting the best, and applying a random mutation to generate a child model hyperparameter configuration (lines [16][17][18]. The generated architecture and hyperparameter configurations are concatenated and sent for evaluation. Note that in the beginning of the search, the population queue does not have number of finished evaluations (given that all evaluations do not necessarily finish in the same time). Therefore, the architecture configurations are generated at random while the population size is smaller than (line 20) The mutation corresponds to choosing a different operation for one variable node in the search space. This is achieved by first randomly selecting a variable node and then choosing (again at random) a value for that node excluding the current value. Then, the child is added to the population by replacing the oldest member of the population. The BO component of AgEBO optimizes the hyperparameters (ℎ ) by marginalizing the architecture decision variables (ℎ ). The BO method generates hyperparameter configurations as follows. It starts by sampling a large number of unevaluated hyperparameter configurations. For each sampled configuration ℎ , it uses a model to predict a point estimate (mean value) (ℎ ) and standard deviation (ℎ ). The sampled hyperparameter configurations are ranked by using the upper-confidence bound (UCB) acquisition function, an optimistic policy [31] which consider the best case scenario in case of uncertainty: where ≥ 0 is a parameter that controls the trade-off between exploration and exploitation. A value of = 0 corresponds to pure exploitation, where the hyperparameter configuration with the lowest mean value is always selected. A large value of corresponds to exploration, where hyperparameter configurations with large variance are selected. Evaluation of such configurations results in improvement of the model . A typical BO optimization method with UCB is sequential and generates only one hyperparameter configuration at a time. This is not useful in our setting given the scale required by the AgE method. Therefore, to generate multiple hyperparameter configurations at the same time, we adopt an asynchronous BO that leverages multipoint acquisition function based on a constant liar strategy [15]. This approach starts by selecting a hyperparameter that maximizes the UCB function. The model is retrained with the selected hyperparameter configuration and a dummy value (lie) corresponding to the min value of collected objectives. The next hyperparameter configuration is obtained by maximizing the UCB function using the updated model. The process of selecting a configuration and retraining the model with a lie is repeated until the required number of configurations are sampled. The mean of all the validation accuracy values found up to that point is used as a lie. While several sophisticated asynchronous BO methods exist, the adoption of the constant liar strategy is motivated by its computational simplicity and low overhead. Since the mutation operation in AgE method is simple, the BO method needs to generate multiple configurations in short computation time. Failure to do so will adversely affect the overall node utilization. This approach is motivated by the fact that we have more variables to optimize in the neural architecture space than in the hyperparameter space. Therefore, we can take advantage of the efficient sampling of BO without exploding the number of dimensions, which slow down the frequent refit of the surrogate model.

Implementation details
We implemented AgEBO in DeepHyper 1 [5], open-source scalable AutoML software designed for neural architecture and hyperparameter search. A high-level implementation overview of the AgEBO method is shown in Figure 10. We integrated the Ray Python package [24] within DeepHyper to schedule the evaluation of architectures concurrently. Specifically, at the beginning of each run, a cluster of Ray workers is launched. One of the nodes is called head node because it centralises the initial connections of all other nodes called workers. Each worker is launched with a set of available resources of 8 CPUs and 8 GPUs because Ray will enforce the isolation of available resources when tasks are submitted. Therefore if we define tasks which use 1 CPU and 1 GPU per task, we can 1 https://github.com/deephyper/deephyper place 8 tasks on this worker. Once the cluster is launched, we run our Algorithm 1 from the head node and call this process driver which connects to the Ray cluster. The function which evaluates architectures is exported as a Ray remote function with a maximum number of calls set to 1 to enforce a fresh restart of workers at each call and free properly the GPU memory reserved by Tensorflow.
The evaluation function is also defined with a number of resources = ( , ) needed for its execution. This number can vary in our study. For example, to train a neural network on 4 GPUs, we will set = (4, 4) to the evaluation function. Therefore, the submit_evaluation interface of AgEBO asks the Ray cluster to launch a task which is responsible for running the architecture training on resources, collecting the validation accuracy values, and returning the results through a get_finished_evaluations interface. All GPUs present on the head node are still available for computation.

EXPERIMENTS
We used two large tabular data sets from the ECP-CANDLE project [33]: Combo and Attn. The selection was motivated by the unique nature of pharmacogenomic data. Both benchmarks consider the drug response problem, i.e., predicting the activity of a drug treatment against a cancer cell line, a critical step toward precision oncology. To accomplish this task, a host of diverse data types are used as input features. They include molecular assays such as protein, microRNA and gene expression profiles as well as drug descriptors and fingerprints. These tabular data modalities do not readily lend themselves to conventional deep learning architectures such as convolution. Yet, their intrinsic biological and chemical structures make them good candidates for mining inductive prior in model search. While Combo benchmark models combinational drug response in a regression problem, and Attn is a single drug response classifier. Together, they include all five of the aforementioned tabular feature types.
(1) Combo [34] benchmark contains 220,890 data points in the training set and 55,222 data points in the testing set, 3 inputs with 942, 3,839 and 3,839 features respectively and 1 regression output. Compressed it takes about 4.2 GB in total.
The task is to predict the growth percentage given a cell line molecular features and the descriptors of two drugs. We use 20% of the training as a validation set because none is provided.
(2) Attn [9] benchmark contains 271,915 data points in the training set, 33,989 in the validation and testing sets, 1 input with 6,212 features and 2 classes. Compressed it takes about 7.9 GB in total. The task is to classify the drug response in two classes. The distribution of classes in the data sets is unbalanced therefore we use the following class weights to mitigate this effect during learning: {0: 0.52, 1: 13.87}.
For the Combo dataset, the baseline model is composed of two submodels followed by a final processing step. The first sub-model takes the cell expression as input. It is then composed of three dense layers each with 1,000 units and ReLU activation. The second sub-model has the same architecture, but it is used to process the two drug descriptors (i.e., weight sharing). Then the output of these submodels is concatenated and input in three dense layers with 1,000 units and ReLU activation to finally arrive to the output layer. The other hyperparameters are: mean squared error (MSE) for the loss, a batch size of 32, a learning rate of 0.01 and the optimizer is Adam.
For the Attn dataset, the baseline model is an attention-based neural network composed of 8 hidden dense layers of size [1000, 1000, 1000, 500, 250, 125, 60, 30] with ReLU activation function except for the third layer which has softmax activation due to the self-attention mechanism. Each of the dense layers is followed by batch normalization and dropouts of rate 0.2 are also used from the fourth layer. The other hyperparameters are: categorical crossentropy for the loss, a batch size of 32, a learning rate of 10 −5 , the optimizer is SGD. A callback to reduce the learning rate on plateau is used with a patience of 40 and a factor of 0.2 while monitoring the validation AUROC of the model. A second callback for early stopping is also monitoring the validation AUROC and has a patience of 200.
To avoid overfitting, we did not use the test data set during the search. For every network generated during the search, the training data was used to train the model and the validation data was used to evaluate the accuracy. At the end of the search, we selected the best network based on the objective found at the last epoch of training, retrained it on the original training data, and evaluated its accuracy on the test data.
Experiments were run on the ThetaGPU cluster at the Argonne Leadership Computing Facility (ALCF). ThetaGPU comprises 24 NVIDIA DGX A100 nodes, each equipped with eight NVIDIA A100 Tensor Core GPUs, two AMD Rome CPUs of 64 cores, 320 GB of GPU memory and 1 TB of DDR4 memory. Since there are already 8 GPUs per node, we did not consider the data-parallelism that spans across multiple nodes. Instead, the data-parallel training within AgEBO is limited to a single node; however, it uses multiple GPUs within the single node to accelerate training. To optimise the training time and reduce overheads we cache the data in the DDR4 memory of each node. Whenever there are multiple evaluation tasks per node, they all access the data from the cache (no data set replication within each node). We use the cache and prefetch methods from tensorflow.dataset, the prefect is set with AUTOTUNE. CUDA and CUDNN 11.0 are used jointly with NCCL 2.7.8. Finally, we activate the XLA compilation with TF_XLA_FLAGS=-tf_xla_enable_xla_devices.
By default, the NAS experiments were run for a wall time of 3 hours on 8 nodes of ThetaGPU. One process on the head node was reserved for the search, and all 64 GPUs were used as workers to train and validate the models generated by the search methods.
AgE was used as the baseline. The optimizer was set to Adam [? ], and each model was evaluated with a maximum of 100 epochs of training. The linear scaling rule [17] was employed to scale the batch size and learning rate with respect to the number of parallel GPUs used for one evaluation. A callback was used to automatically reduce the learning rate on a plateau (ReduceLROnPlateau) with a patience of 5 epochs. An other callback was used to automatically stop the training (EarlyStopping) with a patience of 10 epochs. The objective is to maximise the validation 2 for Combo and Precision-Recall for Attn. For the search, the population ( ) and sample sizes ( ) were set to 100 and 10, respectively. The batch size and learning rate were set to 32 and 0.001 (default values), respectively. AgEBO variants adopt the same training strategy as AgE uses. The difference between AgEBO variants and AgE is that the values of the batch size, learning rate, optimizer, loss, and the two patience can be tuned concurrently along with the architecture search. For AgEBO the range for hyperparameters was set as follows: batch size ( 1 ) ∈ [|16, 2048|]; learning rate ( 1 ) ∈ (0.0001, 0.01), batch size and learning rate are sampled in a log-uniform scale within BO; optimizer ∈ ["sgd", "rmsprop", "adagrad", "adam", "adadelta", "adamax", "nadam"], the patience of ReduceLROnPlateau ∈ (3,30), the patience of EarlyStopping ∈ (3,30). The loss ∈ ["mae", "mse", "huber_loss", "log_cosh", "mape", "msle"] was used as a range for Combo because it was a regression problem. Nevertheless, for the Attn, the loss was fixed to categorical cross-entropy. This is because we enforced exclusivity in the class label prediction using the softmax activation in the output layer with one-hot encoded target.

Impact of hyperparameter tuning
Here, we show that hyperparameter tuning with BO can significantly improve the accuracy of the neural architecture search. Moreover, we show that the accuracy of the architectures discovered by the AgE method with naive data-parallel training deteriorates significantly. Nevertheless, hyperparameter tuning in AgEBO circumvents this issue.
First, we evaluated AgE and AgEBO without data-parallel training. Next, we used data-parallel training, where we varied the number of GPUs per architecture evaluation for data-parallel training and analyzed time to solution and accuracy. We conducted all these experiments on 8 ThetaGPU nodes (default setting). We use {AgE, AGEBO}--to denote a variant, where and are the number of GPUs per architecture evaluation and the number of nodes, respectively. For example, AgE-1-8 refers to AgE ran with 1 GPU per architecture evaluation with 8 nodes. We used the default learning rate and batch size for AgE-1-8. Note that as we vary the number of GPUs per evaluation from 1, 2, 4, and 8, the number of simultaneous architecture evaluations become 64, 32, 16, and 8, respectively. In AgE-2-8, AgE-4-8, AgE-8-8, batch size and learning rate for different numbers of GPUs were scaled using the linear scaling rule. In AgEBO variants, all the hyperparameters are tuned using BO.    The comparison of AgE-1-8 and AgEBO-1-8 shows that the hyperparameter tuning with BO significantly improves the search to find high-performing architecture in a short time. AgE-1-8 reaches a validation 2 of 0.923 around 100 minutes whereas AgEBO-1-8 reaches that accuracy within 30 minutes and a validation 2 of 0.938 at the end.
From the results of AgE-2-8, AgE-4-8, AgE-8-8, we can see that the naive data-parallel training significantly affects the AgE's ability to find architectures with high accuracy. The training time distribution computed from all architectures found by the search shows a significant reduction in training time from 2 to 4 GPUs, but the speedup from the use of 8 GPUs is negligible. We did not observe a significant reduction in training time from 1 GPU to 2 GPUs. This is because AgE-1-8 was run without Tensorflow distributed library but AgE-2-8 (AgE-4-8 and AgE-8-8) incurs the overhead of using it. We can also observe that the number of architectures explored by AgE reduces with an increase in the number of GPUs per architecture evaluation. All these observations clearly establish that the naive data-parallel training degrades the performance of AgE.
The results of AgEBO-2-8, AgEBO-4-8, AgEBO-8-8 that have autotuned data-parallel training show that they reach similar accuracy for different number of GPUs per architecture evaluation. AgEBO-2-8, AgEBO-4-8, AgEBO-8-8 reach the accuracy of AgE-1-8 within 40 minutes of search. There is a slight reduction in accuracy (to the third decimal place) as we increase the number of GPUs, but the reduction is not as drastic as seen with AgE variants. For the same number of GPUs per architecture evaluation, AgEBO achieves better validation 2 than that of AgE. These results show that the joint optimization of hyperparameters and neural architectures is able to circumvent the issues posed by the naive data-parallel training. It is interesting to note that despite the significant reduction in the number of architectures evaluated, AgEBO counters the loss of accuracy by tuning the hyperparameters, which eventually leads to a similar training time distribution. AgE alone cannot achieve this with the fixed hyperparameter values during the search.

Comparison with the manually-designed baseline
Here, we show that the best models found by AgEBO outperform the manually designed neural network baselines for Combo and Attn with respect to accuracy and training time.
For the comparison, we selected the best model obtained by AgEBO and retrained the model on a single ThetaGPU node with a wall-time of 1 hour and a maximum of 100 epochs. We also included the best model found by AgE-1-8 (AgE without data-parallel training) for comparison.
The results are shown in Tables 2a and 2b. From the results, we can observe that AgEBO variants outperform both Combo and Attn baselines with respect to all the metrics. We observe that there is a slight decrease in the accuracy values by using the best found models from AgEBO-2-8, AgEBO-4-8, and AgEBO-8-8. For both data sets, AgEBO discovers network architectures with large number of trainable parameters to increase the accuracy but at the same time they are much faster to train. For Combo, the best    network found by AgEBO-2-8 is 7.25x faster than the baseline. For Attn, the best network found by AgEBO-2-8 is 40.53x faster than the baseline. The evolution of the validation 2 for a few best performing network is shown in Figure 4a. For Combo, the baseline starts to stagnate and oscillate from 25 epochs. The best network found by AgEBO reaches a high accuracy within 25 epochs. For Attn, which is characterised by heavy class imbalance, the baseline improves slowly and and stops after 70 epochs because of the wall time. The best network found by AgEBO reaches a high accuracy within 20 epochs, oscillates from 20 to 40 trying to improve, and stops without improvement after 50 epochs.
The key advantages of AgEBO in light of multiple GPUs per evaluation are high accuracy and faster training of the best model found during the search. This is evident from the results of Table  3 and 4. Given the same number of nodes, AgEBO-2-8 achieves models with training times that are 1.94x (Combo) and 1.68x (Attn) faster than AgEBO-1-8 (Table 3 and Table 4) without significant loss in accuracy. AgEBO-2-8 starts with 2 GPUs per evaluation, which can result in faster models during the initial iterations of the search but their accuracy values will be low. As the search proceeds, due to the asynchronous nature of AgEBO, the models with high validation accuracy values that train faster are reinforced and have a higher chance of survival in the population. However, further increases in the number of GPUs per evaluation do not result in significant reduction in training time for these data sets. To offset the loss of accuracy in AgEBO-4-8 and AgEBO-8-8, AgEBO generates models that take longer to train (more epochs to reach similar accuracy). The comparison between AgE-1-8 and AgEBO-1-8 shows that even in the absence of data parallel training, optimizing the hyperparameters through BO increases the accuracy and decreases the training time. For Combo and Attn data sets, the test 2 values improve from 0.919 to 0.931 and from 0.970 to 0.995, respectively. The training times of the best models from AgEBO-1-8 are 1.2x and 3.49x faster than that of AgE-1-8. It is worthwhile to mention that faster training and inference models are useful for a number of downstream cancer predictive modelling tasks, such as training a large number of models with the same architecture but with different random seeds for building a robust ensemble with uncertainty quantification and accelerating high-throughput in silico drug pair screening with faster inference.

Scaling
Here, we compare the scaling behaviour of AgE and AgEBO and show that hyperparameter tuning helps AgEBO to achieve better scaling.
All the experiments were run with a wall time of 3 hours and we use 2 GPUs per architecture evaluation. The number of nodes is progressively increased from 1, 2, 4, 8, and 16. Given that in these settings, we have 4 workers per node (8 GPUs for each node with 2 GPUs/evaluation), it results in a minimum of 4 workers (1 node) and a maximum of 64 workers (16 nodes) in parallel. Figure 5 shows the percentage of cumulative time spent in training the generated neural networks normalized by the total available time of GPU computation (i.e., 3 hours × number of GPUs). We observe that minimal time is spent in starting the Ray cluster, loading the data or initialising the model before training. These results show that AgEBO does not have significant overhead to generate networks for evaluation given the node counts considered. Note that these measurements do not take into account the overhead incurred by Tensorflow distributed or gradient synchronisation. Table 3 summarizes the scaling results. For AgEBO, we can observe that increasing the number of nodes increases the number of evaluations and improves the best validation 2 value. More importantly, the time to reach baseline accuracy significantly reduces by increasing the number of nodes. By scaling to 4 and above nodes, AgEBO reaches the accuracy of the baseline with 7 minutes. While the trends are similar for AgE, the number of evaluations, the best validation 2 value, and the time to reach baseline are poorer than those of AgEBO. These results clearly show that BO significantly helps AgEBO in scaling.
The results in Table 3 do not capture the usefulness of the scaling in its entirety. To that end, we analyzed the number of unique architectures obtained by AgE and AgEBO that are better than the baseline over time. This measures the strength of AgE and AgEBO to outperform the baseline and how it changes as we increase the number of nodes. The results are shown in Figure 6. These results clearly show that AgEBO and AgE take advantage of large number of nodes to find architectures that are better than baseline. For AgEBO, we found a linear scaling for up to 8 nodes: the number of unique architectures that are better than baseline almost doubles by doubling the number of nodes. At 16 nodes, there is a drop in linear scaling. Furthermore, we can observe that AgEBO completely dominates AgE. The best AgE variant, AgE-2-16 obtains 300 unique architectures that are better than baseline after three hours. The best AgEBO variant, AgEBO-2-16 reaches the same number in 75 minutes and in three hours it found more than 800 architectures that are better than the baseline.

Comparison with mixed BO and mixed AgE
In AgEBO tabular, the BO component of AgEBO optimizes the hyperparameters by marginalizing the architecture decision variables.
Here, we analyze if this is an effective strategy when compared to BO that optimizes hyperparameters along with architecture decision variables and AgE that optimizes architecture decision variables along with hyperparameters. We refer these two methods as mixed BO and mixed AgE, respectively. In mixed BO, we used the same asynchronous Bayesian optimization based on LCB acquisition function and with the same = 0.001. In mixed AgE, we concatenate the list of hyperparameters and neural architecture discrete dimensions. At each iteration of AgE, one of these dimensions is picked at random and the mutation corresponds to a sampling from the prior distribution of this 0 0. 5 Table 3: Scaling the number of nodes with 2 GPUs per architecture evaluation for AgE and AgEBO. A "-" denotes that the condition was not met. The results are shown in Figure 7a. We can observe that AgEBO-4-8 and mixed AgE-4-8 outperform mixed BO-4-8 with respect to validation 2 values. While AgEBO and mixed AgE reach 0.931, BO stagnates at 0.925. Figure 7b shows the number of unique architectures that are better than the baseline found by the three methods over time. We can observe that AgEBO outperforms both mixed AgE and BO. In 15 minutes, AgEBO achieves the number that is better than both mixed AgE and BO and it obtains more than 200 unique architectures that are better than the baseline.
The effectiveness of marginalization in AgEBO can be attributed to the implicit regularization mechanism of the AgE method and a stronger exploitation of the BO method. The AgE's regularization mechanism favours architectures that retrain well across multiple generations. The only way architecture can stay in the population for a long time is through inheritance-it should be passed down from parent to child through multiple generations. Whenever an architecture is inherited it will undergo retraining. If the retraining validation accuracy becomes low, the architecture will be removed from the population. Given that BO has a stronger exploitation due to small value, the hyperparameter values generated will be biased towards the hyperparameter values of the previously obtained high-performing architectures in the population. If any of the high-performing architectures becomes low-performing after inheritance and retraining, it will be removed from the population. Consequently, only the architectures that result in the improvement of the validation accuracy after inheritance and retraining are allowed to evolve in the population during the search.

Exploration and exploitation in AgEBO
Here, we study the effect of exploration and exploitation of BO within AgEBO by varying values. We show that stronger exploitation is critical for the effectiveness of AgEBO. The value in Eq. 3 controls the trade-off between exploration and exploitation in BO. In addition to the default value of 0.001, we ran AgEBO with six values: {0, 0.001, 0.01, 0.1, 1.96, 19.6}. Note that 1.96 is the typical value in Scikit-Optimize, which provides a balance between exploration and exploitation. The value of 19.6 is selected to enforce large exploration. On the other hand, the value of 0 is to enforce pure exploitation where the variance in Eq. 3 is totally ignored. The values of {0.01, 0.1} enforce different degrees of exploitation. We ran the experiments on the Combo data set. Figure 8a shows the best validation 2 found by the search over time. Pure exploitation ( = 0), balance between exploration and exploitation (default = 1.96) large exploration ( = 19.6) did not result in high accuracy. Figure 8b shows the number of architectures performing better than the baseline found by AgEBO for different values. We can observe that, AgEBO with the value of {0.001, 0.01, 0.1} (stronger exploitation) outperforms those with 1.96 (balance between exploration and exploitation) and 19.6 (stronger During the iterative phase, given the stronger exploitation setting, hyperparameter configurations are generated close to the best ones found so far in the search. On the other hand, there is a significant degree of exploration with values of 1.96 and 19.6. This results in a lot more low performing configurations of hyperparameters. As discussed in Section 4.4, the AgEBO's effectiveness is attributed to the implicit regularization mechanism of AgE method and a stronger exploitation of in the BO method. When the BO method have stronger exploration, it generates hyperparameter configurations that are different from the previously best performing hyperparameter values. When these values are used for the high-performing architectures after inheritance and retraining, they will most likely become low performing ones and removed from the population. Therefore BO should generate hyperparameter values that are close to the previously found ones. However, it should be noted that a pure exploitation ( = 0) will keep generating the same hyperparameter values after a few generations, which did not help AgEBO to generate high-performing architectures.

Synchronous vs Asynchronous AgEBO
Compared to synchronous BO, asynchronous methods have received relatively less attention in the BO community. Therefore, we compare synchronous and asynchronous BO with constant liar strategy in the AgEBO to show that the latter is effective. The synchronous version of AgEBO is obtained by placing a synchronisation barrier after line 9 ( get_finished_evaluations()) in Algorithm 1. Specifically, AgEBO proceeds only after getting the evaluation results. For this experiment, we used AgEBO-2-8 (2 GPUs per evaluation and 8 nodes) on the Combo data set. The synchronous and asynchronous versions are referred to as AgEBO-2-8 sync and AgEBO-2-8 async, respectively. Figure 9 shows the number of unique architectures found by AgEBO-2-8 sync and AgEBO-2-8 async that are better than the baseline, respectively. These results show that the asynchronous approach is better than the synchronous one. Both synchronous and asynchronous outperforms baseline in about 6 mins. However, we observed that AgEBO-2-8 async has a much better node utilization 95% whereas it was only 26% for AgEBO-2-8 async. Consequently, the number of evaluations performed on AgEBO-2-8 async is 4.77x more than AgEBO-2-8 sync (759 for the former, 159 for the latter). The best 2 value found by AgEBO-2-8 async was 0.936 whereas it was 0.926 for AgEBO-2-8 sync. While a few sophisticated asynchronous BO methods exist in the literature [3,4,21], they seek to achieve balance between exploration and exploitation. Nevertheless, such methods are not required in our setting because of the synergy between AgE and BO with stronger exploitation.

RELATED WORK
From the novelty perspective, our method has three components: hyperparameter search for data-parallel training, joint NAS and HPS, and application to tabular data. We review the related work from the perspective of each component.
The literature on HPS for tuning hyperparameters on distributed data-parallel training to optimize learning rate, batch size, and others is limited. A commonly used approach to adapt learning rate and batch size in distributed data-parallel training is the linear scaling rule. The values of the learning rate and batch size used for the single-process training are multiplied by the number of processes in distributed data-parallel training. In an Amazon blog [1], the importance of tuning learning rate and batch size for a given number of GPUs in data-parallel training has been discussed. Specifically, the Amazon SageMaker HPO tool has been used as a proof of concept; but the study was not performed at scale, and the effectiveness was not assessed on wide range of data sets. The use of BO to tune the learning rate, batch size, and other hyperparameters while using distributed training has never been investigated before.
Within NAS, several approaches have been proposed to reduce the training time. Examples include using smaller architecture search and stacking [40,41], reducing the number of epochs [35], computing the validation performance from a randomly initialised DNN [36], estimating the accuracy performance of DNN for a large budget (time) when trained with a smaller budget [37], sharing the weights of previously trained DNN [26], imposing a time budget [6], and using information from data relatively to an initialised DNN (but only for convolution NN) without training [23]. These methods have several limitations. Stacking the simpler model is feasible for image data sets but can lead to overfitting in tabular data sets; and reducing the epochs and time budget during NAS can result in a low performing model [35]. Compared with these methods, data-parallel training is a promising approach because of its ability to match with the learning curve of the classical training while consequently speeding up the training [17]. Nevertheless, the use of data-parallel training within NAS was not investigated before.
The joint NAS and HPS approach that we propose is similar to BO Hyperband (BOHB) [35]. It considers the joint space and uses a multivariate kernel density estimation model to sample promising configurations. The sampled configurations are evaluated by using a successive halving approach, where promising configurations are allowed to run longer with more resources. Compared to our approach, BOHB does not 1) differentiate the model hyperparameters from neural architecture hyperparameters; and 2) utilize data-parallel training to speedup the evaluation of neural networks, but instead adopt successive halving to allocate more resources to promising configurations. While it can be quite effective under limited resource setting, scaling the successive halving method can lead to poor node utilization.
AutoML for tabular data has received considerable attention in recent years. Notable examples include auto-sklearn [12], Auto-WEKA [32], H2O AutoML [19], and TPOPT [25]. A benchmark [14] of these methods was conducted to compare their performance on different data sets. The auto-sklearn approach proved more robust in general. We tested auto-sklearn on our data sets but the results were poor. Recently, AutoGluon [10] and Auto-PyTorch [39] have emerged as new AutoML methods for tabular data. AutoGluon uses an ensemble of many different learning algorithms to boost their performance. Auto-PyTorch also uses an ensemble approach, but the models are restricted to DNNs. As an exploratory study, we used four large tabular data sets from the OpenML benchmark [13]. The selection was motivated by a tabular data benchmark study using AutoGluon. Among all the data sets benchmarked with AutoGluon, we selected the four largest data sets (Covertype [20]:, Airlines [2], Albert [18], and Dionis [18]) having the largest number of data points. We found that the prediction accuracy of AgEBO is better than or comparable to that of AutoGluon and Auto-PyTorch and provides a significant advantage with respect to the inference time.
We developed AgEBO-Tabular, a joint neural architecture (NAS) and hyperparameter search (HPS) method to discover high-performing neural network models for tabular data. We developed an architecture search space for generating fully connected neural networks with skip connections and possibly with multiple inputs. The search method combines two distinct methods: (1) aging evolution (AgE), a parallel NAS to search over the architecture decision variables; and (2) an asynchronous Bayesian optimization (BO) method to automatically tune the hyperparameters of data-parallel training in order to reduce the evaluation time of each architecture.
We showed that using data-parallel training in AgE without tuning the learning rate, batch size, and other hyperparameters can adversely affect the accuracy. Then, we demonstrated that AgEBO can improve the accuracy of the discovered models and the time to generate high-performing neural networks. We compared the bestdiscovered models from AgEBO with state-of-the-art human-made models and showed that AgEBO outperforms the state-of-the-art human made models both in accuracy and training time.
The analysis of the best values obtained by AgEBO showed the need for data-set-specific tuning. Moreover, we showed that, unlike the typical BO that balances exploration and exploitation, a stronger exploitation is critical for AgEBO for generating high-performing models in short computation time.
The algorithmic components (AgE and BO) of the proposed AgEBO method are not specific to NVIDIA accelerator model. We can generalize and adapt AgEBO on CPU based clusters or other CPU+accelerator type platforms. Moreover, the only component that is specific to the tabular data set is the search space. We will expand the search space for image, text, and graph data and evaluate the effectiveness of the proposed AgEBO method.
Our other future work will include (1) applying AgEBO to generate neural architectures for other data types such as images, texts, and graphs; (2) developing multinode data-parallel training within NAS for larger data sets; (3) developing meta-learning and transfer learning approaches to reuse the knowledge and results from previous experimental runs for related data sets; and (4) adaptive and flexible workflow and scheduler system for joint model and resource optimization.
The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory ("Argonne"). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan. http: //energy.gov/downloads/doe-public-access-plan We conducted additional set of experiments on ALCF's Theta supercomputer, a CPU-based system using 4 tabular datasets from the OpenML [13] benchmark. These experiments were based on an implementation of data parallelism using MPI, Horovod, and the warmup learning strategy.
A.1 Implementation details Fig. 10 shows a high-level overview of the implementation in CPU-based system. Algorithm 1 ran on a single process P. Deep-Hyper leveraged the Balsam workflow system [29] to schedule the evaluation of architectures concurrently. Specifically, the sub-mit_evaluation interface of AgEBO calls the Balsam workflow system, which is responsible for running the architecture training on workers (via mpirun), collecting the validation accuracy values, and returning the results through a get_finished_evaluations interface. We allocated one compute node for the search. We used the Horovod library [30] for the distributed data-parallel training implementation within AgEBO. The AgEBO- Tabular

A.2 Experiments
We used four tabular data sets from the OpenML [13] benchmark. The selection was motivated by a tabular data benchmark study using AutoGluon [10], a recently proposed state-of-the-art AutoML method for tabular data. Among all the data sets benchmarked with AutoGluon, we selected the following four largest data sets having the largest number of data points: (1) Covertype [20]: It contains 581,012 data points, 54 input features, and 7 classes. The task is to predict the forest cover type given cartographic variable input data. (2) Airlines [2]: It contains 539,383 data points, 8 input features, and 2 classes. The task is to develop a model to indicate whether a given flight will be delayed or not given input data of the scheduled departure. For each data set, we grouped the data for training, validation, and testing as in the Auto-PyTorch benchmark study. Specifically, we used 42% for training, 25% for validation, and 33% for testing. In all the AutoML methods, we used the training and validation data set within AgEBO-Tablular. The selected best model was evaluated on the testing data. Experiments were run on the Theta supercomputer at the Argonne Leadership Computing Facility (ALCF). Theta is a Cray XC40 11.69-petaflops system composed of 4,392 nodes with Intel Knights Landing CPUs of 64 cores each equipped of 192 GB of DDR4 memory. Since the data set that we consider fits in a single-node memory, we did not utilize multinode data-parallel training. Instead, the dataparallel training within AgEBO was limited to single node; however, it uses multiple processes within the single node to accelerate training.
The number of threads per process within the single node, , is set to the ratio of the number of threads per node, , and the number of process per node, . The threading is configured based on guidelines provided by the ALCF, which is based on Ten-sorFlow documentation: intrathreads = OMP_NUM_THREADS = ; interthreads = 2; CPU affinity = depth (equivalent to: KMP_AFFINITY = "granularity=fine,verbose,compact,1,0"); KMP_BLOCK_TIME = 0.
By default, the NAS experiments were run for a wall time of 3 hours on 129 nodes of Theta. One node was reserved for the search, and 128 nodes were used as workers to train and validate the models within AgEBO.
AgE was used as the baseline. The optimizer was set to Adam [22], and each model was evaluated for 20 epochs of training. A gradual warmup strategy [17] was employed for the first 5 epochs. A callback was used to automatically reduce the learning rate on a plateau with a patience of 5 epochs. The objective in the AutoML methods is to maximize the validation accuracy. For the search, the population ( ) and sample sizes ( ) were set to 100 and 10, respectively. The batch size and learning rate were set to 256 and 0.01, respectively. AgEBO variants adopt the same training strategy as AgE uses. The difference between AgEBO variants and AgE is that the values of the batch size, learning rate, and number of processes for data-parallel training can be tuned concurrently along with the architecture search.
A.2.1 Impact of static data-parallel training on AgE. We show that the accuracy of the architectures discovered by the AgE method with data-parallel training deteriorates significantly without tuning the learning rate, batch size, and number of processes.
We evaluated AgE with data-parallel training without BO but varied the number of processes. We used the default learning rate and batch size for = 1. The learning rate and batch size for different numbers of processes were scaled by using the linear scaling rule. We ran the experiments on the Covertype data set.
The results are shown in Figure 11 and Table 4, where AgErefers to AgE with processes for data-parallel training. From the results we observe that increasing the number of ranks from 1 to 4 per evaluation increases the accuracy. This increase can be attributed to the reduced training time for architecture evaluation, which increases the number of evaluated architectures from 632 to 2,421. Nevertheless, for AgE-8, we observe that the accuracy significantly decreases despite the large number (4,221) of evaluated architectures. The poor accuracy of AgE-8 can be attributed to the scaled learning rate and batch size values for 8 processes and/or the possibility that 8 is not the right value for achieving reduction in training time without losing accuracy.    A.2.2 Impact of autotuned data-parallel training within AgEBO.
Here we show that tuning the learning rate, batch size, and number of processes through BO improves both the accuracy and time to solution.
To analyze the effectiveness of BO within AgEBO, we compared it with two of its variants. AgEBO-8-LR and AgEBO-8-LR-BS. In the former, only the learning rate was tuned by setting the batch size and the number of processes for the data-parallel training to the default batch size and 8, respectively. In the latter, the batch size and learning rate were tuned by setting the number of processes to 8. As a baseline, we used AgE-8. The experiments were run on the Covertype data set.
The results are shown in Figure 12. We observe that the AgEBO variants outperform AgE-8 with respect to both accuracy and the time to reach that accuracy. The comparison between AgEBO-8-LR and AgE-8 shows that tuning the values of the learning rate leads to significant improvement with respect to both accuracy and time to solution. Similarly, AgEBO-8-LR-BS achieves a higher accuracy value than that of AgEBO-8-LR within a shorter time. However, AgEBO, which tunes all three hyperparameters, outperforms AgEBO-8-LR-BS. An exception is in the initial phases of the search (first 30 minutes), which is due to the initial rank exploration of AgEBO and its impact on the training time. Specifically, this can be attributed to the exploration of different parallelism settings during that phase, which increases the evaluation time of the architectures.
To ensure that the observed superior accuracy of AgEBO is not by chance, we analyzed the number of unique architectures found over time that have a validation accuracy higher than 0.90 for AgE-variants and AgEBO. The threshold of 0.90 is computed by taking the minimum of 0.99 quantiles of validation accuracy for each variant. The results are shown in Figure 13. We observe that AgEBO obtains a larger number of high-performing architectures than that of AgE-variants. Moreover, despite given the same number of nodes, AgEBO is twice as fast as AgE-variants in reaching the same number of high-performing architectures. Specifically, AgE-4 and AgE-8 obtain 10 2 high-performing architectures in 180 minutes whereas AgEBO obtains the same number within 90 minutes. Figure 13: Number of unique high-performing models obtained by AgEBO and AgE-variants on the Covertype data set.

A.2.3 Comparison with AutoGluon and Auto-PyTorch.
Here we show that the prediction accuracy of our method is better than or comparable to that of the two state-of-the-art AutoML software tools AutoGluon [10] and Auto-PyTorch [39] while reducing the inference time of final models.
The two methods rely on ensemble approaches to boost their prediction accuracy values. AutoGluon combines different supervised learning models such as neural networks, LightGBM, CatBoost, random forest, extra trees, and K-nearest neighbors, the hyperparameters of which are automatically tuned. On the other hand, Auto-PyTorch adopts only neural network models but uses an ensemble strategy to improve the accuracy. We compared AgEBO with AutoGluon and Auto-PyTorch on all four data sets. We used AgE-1 as a baseline.
AutoGluon was run on a single node with a time limit of 4 hours for the call to the fit method to compensate for possible issues with the time estimation performed by the software. The hyperparameter_tune=True and auto_stack=True were set to maximize the accuracy as much as possible. The test accuracy was computed separately by reloading the saved models. Table 5 shows the accuracy values of the best models and the corresponding inference time of AgEBO and AutoGluon. We observe that the test accuracy values of AgEBO and AutoGluon are comparable on all four data sets. However, the key advantage stems from the inference time with the trained model. Given that AgEBO generates a single neural network model, the inference time is between 2.7 and 4.3 seconds. On the other hand, AutoGluon relies on stacking a number of models, resulting in an inference time of about 7 minutes.
For Auto-PyTorch, since we cannot install the software in our ALCF Theta software stack because of software dependency issues, we used the results from the LCBench data base [38], which stores the results of experimental runs of the four data sets. We note, however, that although we used the same proportion of the training, validation, and testing split, the exact data splits were not used, the details of which are not available. Moreover, we did not compare against test accuracy from the ensemble strategy from Auto-PyTorch because we cannot retrieve ensemble strategy results from the LCBench database. Therefore, we focus on comparison with validation accuracy values. Figure 14 shows the comparison between the best validation accuracy values found by AgEBO and Auto-PyTorch. We observe that AgEBO achieves validation accuracy values that are higher than those of Auto-PyTorch within 30 minutes of search time. The differences in the accuracy values can be explained by two factors. First, Auto-PyTorch is not designed to generate a single neural network model but to generate multiple models and combine them using an ensemble strategy to have good accuracy. Second, the architecture space of Auto-PyTorch is restricted to a smaller number of trainable parameters and smaller number of layers.
The comparison between AgE-1 and AgEBO in Figure 14 summarizes the benefits of autotuned data-parallel training. For the Airlines data set, the maximal accuracy found with AgE-1 is 0.647 at 121 minutes, whereas AgEBO finds a greater accuracy after 14 minutes and reaches its maximal accuracy of 0.652 after 163 minutes. For the Albert data set, the maximal accuracy found with AgE-1 is 0.662 at 147 minutes, whereas AgEBO achieves a higher accuracy after 36 minutes and reaches its maximal accuracy of 0.665 after 49 minutes. For Covertype, the maximal accuracy found with AgE-1 is 0.918 at 164 minutes, whereas AgEBO achieves a greater accuracy after 20 minutes and reaches its maximal accuracy of 0.927 after 165 minutes. For the Dionis data set, the maximal accuracy found with AgE-1 is 0.869 at 163 minutes, whereas AgEBO achieves a greater accuracy after 11 minutes and reaches its maximal accuracy of 0.900 after 147 minutes. In summary, AgEBO outperforms the AgE-1 with respect to both accuracy values and time to reach those accuracy values.  Across all four data sets we observed that the node utilization of AgEBO is similar to that of AgE-both reach an average value of ≈94%. This can be attributed to the effectiveness of the asynchronous BO that generates hyperparameter configurations with minimal overhead, which are combined with architecture decision variable values and sent for evaluation with minimal delay. Table 6 shows the best hyperparameters obtained by AgEBO for the top 5 best-performing models on the four data sets. Note that AgEBO finds different hyperparameter configurations for different data sets to accelerate data-parallel training. Within the same data set, the hyperparameter configurations obtained for the best models are similar. These results demonstrate the need for data-set-specific hyperparameter tuning for data-parallel training, which is enabled by AgEBO.

AgEBO
We visualized the top 1% configurations based on the validation accuracy values obtained on all four data sets using principal component analysis. This is done by projecting the 37 architecture decisions and 3 hyperparameters of the top 1% configurations into two dimensions, respectively. The results are shown in Figure 15. From the results we can see a similar pattern. Each data set requires different values for architecture decision variables and data-parallel training hyperparameters.

A.2.4 Exploration and exploitation in AgEBO.
Here we study the effect of exploration and exploitation of BO within AgEBO by varying values. We show that stronger exploitation is critical for the effectiveness of AgEBO.
The value in Eq. 3 controls the trade-off between exploration and exploitation in BO. In addition to the default value of 0.001, we ran AgEBO with two values: {1.96, 19.6}. Note that 1.96 is the typical value in Scikit-Optimize, which provides a balance between exploration and exploitation. The value of 19.6 is selected to enforce large exploration. We ran the experiments on the Covertype and the Dionis data sets. Figure 16 shows the number of high-performing architectures found by AgEBO for three different values. The threshold was computed by computing 99% quantiles of the validation accuracy values for the three variants and taking the smallest value. We observe that for both data sets, AgEBO with the default value of 0.001  Table 6: Data-parallel training hyperparameter values obtained by AgEBO for the top 5 best models on the four data sets.
(stronger exploitation) completely outperforms those with 1.96 (balance between exploration and exploitation) and 19.6 (stronger exploration) with respect to the number of high-performing architectures (between one and two orders of magnitude) and time needed to reach the number of the other two variants (between 2x and 3x faster). The exploration of hyperparameter values in AgEBO with a value of 0.001 happens only in the random initialization phase. During the iterative phase, given the stronger exploitation setting, hyperparameter configurations are generated close the best ones found so far in the search. On the other hand, there is a significant degree of exploration with values of 1.96 and 19.6. This results in increased data-parallel training time, which eventually reduces the generation of number of high-performing architectures.