Winning solutions and post-challenge analyses of the ChaLearn AutoDL challenge 2019

This paper reports the results and post-challenge analyses of ChaLearn's AutoDL challenge series, which helped sorting out a profusion of AutoML solutions for Deep Learning (DL) that had been introduced in a variety of settings, but lacked fair comparisons. All input data modalities (time series, images, videos, text, tabular) were formatted as tensors and all tasks were multi-label classification problems. Code submissions were executed on hidden tasks, with limited time and computational resources, pushing solutions that get results quickly. In this setting, DL methods dominated, though popular Neural Architecture Search (NAS) was impractical. Solutions relied on fine-tuned pre-trained networks, with architectures matching data modality. Post-challenge tests did not reveal improvements beyond the imposed time limit. While no component is particularly original or novel, a high level modular organization emerged featuring a"meta-learner","data ingestor","model selector","model/learner", and"evaluator". This modularity enabled ablation studies, which revealed the importance of (off-platform) meta-learning, ensembling, and efficient data management. Experiments on heterogeneous module combinations further confirm the (local) optimality of the winning solutions. Our challenge legacy includes an ever-lasting benchmark (http://autodl.chalearn.org), the open-sourced code of the winners, and a free"AutoDL self-service".


INTRODUCTION
The AutoML problem asks whether one could have a single algorithm (an AutoML algorithm) that can perform learning on a large spectrum of tasks with consistently good performance, removing the need for human expertise (in defiance of "No Free Lunch" theorems [1], [2], [3]). Our goal is to evaluate and foster the improvement of methods that solve the AutoML problem, emphasizing Deep Learning approaches. To that end, we organized in 2019 the Automated Deep Learning (AutoDL) challenge series [4], which provides a reusable benchmark suite. Such challenges encompass a variety of domains in which Deep Learning has been successful: computer vision, natural language processing, speech recognition, as well as classic tabular data (feature-vector representation).
AutoML is crucial to accelerate data science and reduce the need for data scientists and machine learning experts. For this reason, much effort has been drawn towards achieving true AutoML, both in academia and the private sector. In academia, AutoML challenges [5] have been organized and collocated with top machine learning conferences such as ICML and NeurIPS to motivate AutoML research in the machine learning community. The winning approaches from such prior challenges (e.g. auto-sklearn [6]) are now widely used both in research and in industry.
More recently, interest in Neural Architecture Search (NAS) has exploded [7], [8], [9], [10], [11]. On the industry side, many companies such as Microsoft [12] and Google are developing AutoML solutions. Google has also launched various AutoML [13], NAS [14], [15], [16], [17], and meta-learning [18], [19] research efforts. Most of the above approaches, especially those relying on Hyper-Parameter Optimization (HPO) or NAS, require significant computational resources and engineering time to find good models. Additionally, reproducibility is impaired by undocumented heuristics [20]. Drawn by the aforementioned great potential of AutoML in both academia and industry, a collaboration led by ChaLearn, Google and 4Paradigm was launched in 2018 and a competition in AutoML applied to Deep Learning was conceived, which was the inception of the AutoDL challenge series. To our knowledge, this was the first machine learning competition (series) ever soliciting AutoDL solutions. In the course of the design and implementation we had to overcome many difficulties. We made extensive explorations and revised our initial plans, leading us to organize a series of challenges rather than a single one. In this process, we formatted 66 datasets constituting a reusable benchmark resource. Our data repository is still growing, as we continue organizing challenges on other aspects of AutoML, such as the recent AutoGraph competition. In terms of competition protocol, our design provides a valuable example of a system that evaluate AutoML solutions, with features such as (1) multiple tasks execution aggregated with average rank metric; (2) emphasis on any-time learning that urges trade-off between accuracy and learning speed; (3) separation of feedback phase and final blind test phase that prevents leaderboard overfitting. Our long-lasting effort in preparing and running challenges for 2 years is harvested in this paper, which analyses more particularly the last challenge in the series (simply called AutoDL), which featured datasets from heterogeneous domains, as opposed to previous challenges that were domain specific.
The AutoDL challenge analysed in this paper is the culmination of the AutoDL challenge series, whose major motivation is two-fold. First, we desire to continue promoting the community's research interests on AutoML to build universal AutoML solutions that can be applied to any task (as long as the data is collected and formatted in the same manner). By choosing tasks in which Deep Learning methods excel, we put gentle pressure on the community to improve on Automated Deep Learning. Secondly, we create a reusable benchmark for fairly evaluating AutoML approaches, on a wide range of domains. Since computational resources and time cost can be a non-negligible factor, we introduce an any-time learning metric called Area under Learning Curve (ALC) (see Section 2.3) for the evaluation of participants' approaches, taking into consideration both the final performance (e.g. accuracy) and the speed to achieve this performance (using wall-time).
As far as we know, the AutoDL challenges are the only competitions that adopt a similar any-time learning metric.
Acknowledging the difficulty of engineering uni- For domain-specific challenges such as AutoCV, AutoCV2, AutoNLP and AutoSpeech, the challenge results and analysis are presented in [4] and some basic information can be found in Table 1. During the analysis of these previous challenges, we already had several findings that were consistent with what we present in this paper. These include the winning approaches' generalization ability on unseen datasets. As further evidence that we hit the right level of challenge duration and difficulty, 90% of teams found the challenge duration sufficient and 50% of teams  and justify these choices.
The rest of this work is organized as follows. In Section 2, we give a brief overview of the challenge design (see [21] for detailed introduction).  [22].
For the datasets of AutoDL challenge, we are not releasing their identities at this stage to allow us reusing them in future challenges. Some potential uses are discussed in Section 6. However, we summarize their name, domain and other meta-features in Table 2.

Blind testing
Since this was a complete blind evaluation during BOTH phases, we provided additional "public" datasets for practice purposes and to encourage metalearning.
We ran the challenge on the CodaLab platform  This is because all teams adopted a domain-dependent approach and some teams simply used the code of Baseline 3 for certain domains (text in this case).

Metric
The AutoDL challenge encouraged learning in a short time period both by imposing a small time budget of 20 minutes per task and by using an "any-time learning" metric. Specifically, within the time budget, the algorithm could make several predictions (as many as they wanted), along the whole execution. This allowed us to use as performance score the Area under the Learning Curve (ALC): where s(t) is the performance score (we used the NAUC score introduced below) at timestamp t and t is the transformed timẽ Here T denotes the time budget in seconds (e.g.   [24] suggests that average rank is satisfying, compared to other ranking methods, in terms of rank correlation with unseen tasks. We are running similar experiments on AutoDL data and those results hold.

Baseline 3 of AutoDL challenge
As in previous challenges (e.g. AutoCV, AutoCV2, AutoNLP and AutoSpeech), we provided 3 baselines (Baseline 0, 1 and 2) for different levels of use: Baseline 0 is just constant predictions for debug purposes, Baseline 1 a linear model, and Baseline 2 a CNN (see [21] for details  [28], and heavy but more accurate models like LSTM [29] and BERT [30]. They first use light models (such as linear SVC), but the meta-controller switches eventually to other models such as neural networks, with iterative training. If the AUC drops below a threshold or drops twice in a row, the model is switched, or the process is terminated and the best model ever trained is chosen, when the pool is exhausted. with smaller numbers of filters/channels/kernels) and VggVox [33], finally (bidirectional) LSTM [29], with attention mechanism. This strategy allows to make fast early predictions and progressively improves models performance over time to optimize the anytime performance metric.

AUTODL CHALLENGE RESULTS
The The final ranking is computed from the performances on the 10 unseen datasets in the final phase.
To reduce the variance from diverse factors such as randomness in the submission code and randomness of the execution environment (which makes the exact ALC scores very hard to reproduce since the walltime is hard to control exactly), we re-run every submission several times and average the ALC scores.
The average ALC scores obtained by each team are shown in Figure 4 (the teams are ordered by their final ranking according to their average rank). From this figure, we see that some entries failed constantly on some datasets such as frozenmad on Yolo, Kon on Marge and PASA NJU on Viktor, due to issues in their code (e.g. bad prediction shape or out of memory error).
In addition, some entries crashed only sometimes on certain datasets, such as Inspur AutoDL on Tal, whose cause is related to some preprocessing procedure on text datasets concerning stop words. Otherwise, the error bars show that the performances of most runs are stable.

WINNING APPROACHES
A summary of the winning approaches on each domain can be found in Table 3. Another summary using a categorization by machine learning techniques can be found in Table 4. We see in Table 3  workflow shared by almost all teams (see Figure 5).
We note that the module "data" includes not only traditional data of example-label pairs but also metadata, metric, budgets and past performances. These are all potential useful information for meta-learning. based on the meta-data of the current task and poten-tially a meta-dataset of prior tasks (e.g. those provided as public datasets). We believe that this AutoML workflow concisely summarizes the increasingly sophisticated AutoML systems found nowadays and provides the direction for a universal AutoML API design in the future (which is work in progress). This workflow will also be useful for the analysis in Section 5.
The more detailed descriptions for the approaches of the top-3 winning teams and automl freiburg can be found in the Appendix.

POST-CHALLENGE ANALYSES
We carry out post-challenge analyses from different aspects to understand the results in depth and gain useful insights. One central question we ask ourselves is how the components (such as meta-learning, data loading and ensemble), in each approach, affect the speech/times series, text, tabular (and then another cycle in this order). More information on the task can be found in Table 2.

Ablation study
To analyze the contribution of different components in each winning team's solution, we asked 3 teams (DeepWisdom, DeepBlueAI and automl freiburg) to carry out an ablation study, by removing or disabling certain component (e.g. meta-learning, data augmentation) of their approach. We will introduce in the following sections more details on these ablation studies by team and synthesize thereafter.

DeepWisdom
According to the team DeepWisdom, three of the most important components leading to the success of their approach are: meta-learning, data loading and data augmentation. For the ablation study, these components are removed or disabled in the following manner: We carried out experiments on the 10 final phase datasets with the above components removed. The obtained ALC scores are presented in Figure 6. As it can be seen in Figure 6, Meta-Learning can be con-

Combination study
In this section, instead of removing certain components for each winning method, we combine components from different teams. We start from a "base" We focused on the following components, which demonstrated their effectiveness in our previous analyses, including the ablation study:  between the ALC performance and the considered components.
In summary, this limited set of combination experiments did not reveal a significant advantage of mixing and matching modules. The solution of the overall winner DeepWisdom stands out.

Varying the time budget
Up till now, all our experiments are carried out within a time budget of 20 minutes, which may seem rela-

AutoML generalization ability of winning methods
One crucial question for all AutoML methods is whether the method can have good performances on unseen datasets. If yes, we will say the method has AutoML generalization ability. To quantitatively measure this ability, we propose to compare the average rank of all top-8 methods in both the feedback phase and the final phase, then compute the Pearson correlation (Pearson's ρ) of the 2 rank vectors (thus similar to Spearman's rank correlation [43]).
Concretely, let r X be the average rank vector of top teams in the feedback phase and r Y be that in the final phase, then the Pearson correlation is computed The average ranks of top methods are shown in  Table 2, this is a feat from the AutoML community.
Thus it's highly plausible that we are moving one step closer to a universal AutoML solution.

Impact of t 0 in the ALC metric
We recall that the Area under Learning Curve (ALC) is defined by wheret (t) = log(1 + t/t 0 ) log(1 + T /t 0 )     When t0 is large, similar weight is applied on the whole learning curve, performances are uniformly averaged, so being a little bit slow at the beginning is not that bad, and it is more important to have good final performance when the time budget is exhausted (fixed-time learning). The tabular dataset Carla is taken as example. The fact that two learning curves cross each other is a necessary condition for the impact of t 0 on their ranking on this task. Learning curves of top teams on this dataset are shown in 12a. The impact of t 0 on the ALC scores of these curves is shown in 12b. We see that when t 0 changes, the ranking among participants can indeed change, typically the ALC of frozenmad is larger than that of Kon but this is not true for large t 0 . In 12c, the fact that the average rank (over all 10 final phase datasets) varies with t 0 also implies that t 0 can indeed affect the ranking of ALC on individual tasks. However, we see that the final ranking (i.e. that of average rank) is quite robust against changes of t 0 . Very few exceptions exist such as PASA NJU and Inspur AutoDL. Overall, t 0 proved to have little impact, particularly on the ranking of the winners, which is another evidence that top ranking participants addressed well the any-time learning problem.
"meta-learning" and "ensembling": Regarding metalearning, at this stage, it is fair to say that strategies employed are effective, but not very sophisticated.
They rely on pre-selecting off-platform, using pro-

APPENDIX A CHALLENGE PROTOCOL DETAILS
As introduced in Section 2.2, the competition consists of two phases: a feedback phase and a final phase.
Participants should submit an AutoML model code instead of predictions.
Code submitted is trained and tested automatically, without any human intervention. Code submit-  Figure 3 in Figure 14.

APPENDIX C DESCRIPTION OF WINNING METHODS
We present in detail the winning solutions from top-3 winning teams (DeepWisdom, DeepBlueAI and PASA NJU) and the team automl freiburg which made a late submission in the feedback phase but ranked 5th in the final phase. We considered interesting to introduce automl freiburg's approach due to their contributions and for scientific purpose.

C.1 Approach of DeepWisdom (1st prize)
The team DeepWisdom proposed a unified learning framework following a meta-learning paradigm. The framework consists of two parts: meta-train and metainference. The meta-train module takes as input the "public" datasets, which are augmented by the internal data augmentation engine, and the objective function (the ALC metric in the case of the challenge). The meta-trainer generates solution agents, whose objective is to search for best models, using search operators.
In the meta-inference step, a new task is processed taking in one dataset of the challenge. Initial meta-  On the contrary, if training is fast and prediction is slow, the frequency is reduced. This strategy can improve to higher early scores.
In order to improve generalization, multiple light-GBM models are used to make an ensemble with a bagging method.

C.3 Approach of PASA NJU (3rd prize)
The PASA NJU team modeled the problem as three which is an open-source package that uses Bayesian optimization to guide the search of hyperparameters.

C.4 Approach of automl freiburg
In contrast to other teams, automl freiburg adopts a domain-independent approach but focused only the computer vision tasks (i.e. image and video datasets) of this challenge. While for all other tasks automl freiburg simply submitted the baseline to obtain the baseline results, they achieved significant improvement on the computer vision tasks w.r.t. the baseline method. To improve both efficiency and flexibility of the approach, they first exposed relevant hyperparameters of the previous AutoCV/AutoCV2 winner code [54] and identified well-performing hyperparameter configurations on various datasets through hyperparameter optimization with BOHB [36]. They then trained a cost-sensitive meta-model [55] with Auto-Folio [35] -performing hyperparameter optimization for the meta-learner -that allows to automatically and efficiently select a hyperparameter configuration for a given task based on dataset meta-features. The proposed approach on the CV task is detailed next.
First, they exposed important hyperparameters of the AutoCV/AutoCV2 winner's code [54] such as the learning rate, weight decay or batch sizes. Additionally, they exposed hyperparameters for the online execution (which were hard-coded in previous winner solution) that control, for example, when to evaluate during the submission and the number of samples used. To further increase the potential of the existing solution, they extended the configuration space to also include: • An EfficientNet [56] (in addition to kakaobrain's [54] ResNet-18) pre-trained on ImageNet [39]; • The proportion of weights frozen when finetuning; • Additional stochastic optimizers (Adam [57], AdamW [58], Nesterov accelerated gradient [59]) and learning rate schedules (plateau, cosine [60]); • A simple classifier (either a SVM, random forest or logistic regression) that can be trained and used within the first 90 seconds of the submission.
After the extension of the configuration space, they optimized the hyperparameters with BOHB [36] across 300 evaluation runs with a time budget of 300 seconds on eight different datasets (Chucky [61], Hammer [62], Munster [63], caltech birds2010 [64], cifar100 [61], cifar10 [61], colorectal histology [65] and eurosat [66]). These eight datasets were chosen from meta-training data to lead to a portfolio of complementary configurations [67], [68]. Additionally, they added a robust configuration to the portfolio of configurations that performed best on average across the ing choices for training, input pipeline, and architecture) for every task (dataset) in our meta-training set using BOHB [36]. Afterwards, for each dataset i, the best found configuration λ * i is evaluated on the other datasets j ∈ {1, 2, ..., N }, j = i to build the performance matrix (configurations × datasets). For training and configuring the meta-selection model based on performance matrix and the meta-features of the corresponding tasks, the approach uses AutoFolio [35]. At meta-test time, the model fitted by AutoFolio uses the meta-features of the test tasks in order to select a well-performing configuration. eight datasets. Then, they evaluated each configuration of the portfolio for 600 seconds on all 21 image datasets they had collected. In addition, they searched for a tenth configuration (again with BOHB), called the generalist, that they optimized for the average improvement across all datasets relative to the already observed ALC scores. In the end, the meta-train-data consisted of the ALC performance matrix (portfolio configurations × datasets) and the meta-features from the 21 datasets. These meta-features consisted of the image resolution, number of classes, number of training and test samples and the sequence length (number of video frames, i.e. 1 for image datasets). In addition, they studied the importance of the meta features for the meta-learner, and selected an appropriate subset. To optimize the portfolio further, they applied a greedy submodular optimization [68], [69] to minimize the chance of wrong predictions in the online phase. Based on this data, they trained a cost-sensitive meta-model [55] with AutoFolio [35], which applies algorithm configuration based on SMAC [37], [70] to efficiently optimize the hyperparameters of the metalearner. Since the meta-learning dataset was rather small, HPO for the meta-learner could be done within a few seconds. Lastly, they deployed the learned Auto-Folio model and the identified configurations into the initialization function of the winner's solution code.
The workflow of this approach is shown in Figure 15.