We point out important problems with the common practice of using the best single model performance for comparing deep learning architectures, and we propose a method that corrects these flaws. Each time a model is trained, one gets a different result due to random factors in the training process, which include random parameter initialization and random data shuffling. Reporting the best single model performance does not appropriately address this stochasticity. We propose a normalized expected best-out-of-n performance (Boo_n) as a way to correct these problems. Replicating results in deep learning research is often hard. This harms their usefulness to industry, leads to a waste of effort by other researchers, and limits the scientific value of such results.One reason is that many papers provide information insufficient for replication. Details of the experimental setup can significantly influence the results BID13 BID10 BID23 , so the details should be provided at least in appendices, ideally alongside the source code, as was strongly emphasized e.g. by BID17 .However , an important second factor hinders replicability: most deep learning training methods are inherently stochastic. This randomness usually comes from random data ordering in stochastic gradient descent and from random parameter initialization, though there can be additional sources of randomness such as dropout or gradient noise. Consequently, even if we fix the model architecture and the experimental setup (including the hyperparameters), we obtain a different result each time we run an experiment. Statistical techniques are needed to handle this variability. However, in deep learning research, they are heavily underused. What is usually done instead?Most empirical deep learning papers simply report the performance of the best single model (sometimes calling it just "single model" performance). We will later show this is the case at least for some sub-domains. Given the result stochasticity, such method is statistically flawed. The best model performance is not robust under experiment replication, and its expected value improves with an increasing number of experiments performed, among other problems. Since many deep learning publications largely ignore these issues, we dedicate the first part of this article to explaining them in some detail, and later run experiments to quantify them.Appropriate statistical techniques are hence necessary for evaluating (and comparing) the performance of machine learning (ML) architectures. Some well-developed methods exist for such comparisons (a great introduction is given for instance by BID5 ). However, most existing methods focus on comparing the mean performance. This may be one of the reasons why statistical methods are being underused, since mean may be unattractive to researchers in certain situations.There are multiple possible reasons for this. The one that we do consider sound 1 is that when deploying models in practice, it is often natural to train multiple instances of a model and then deploy the best one to production based on a validation set evaluation.2 Underperforming models can be discarded, so the final deployed model does come from 1 Other reasons why researchers resort to the best performance as opposed to the mean may come from the current highly competitive atmosphere in the field with (possibly excessive) focus on performance on standard datasets (see BID4 or BID26 for further discussion), which may motivate researchers to publish only their best results. Also, statistically sound estimation of performance does require repeatedly re-running experiments, which does incur additional cost, which researchers may prefer to invest in additional model tuning, especially in the present situation where reviewers seem not to require statistically sound evaluation of models and on the other hand may favour high-performing models. Of course, these motives should instead give place to effort to do good science, as opposed to a race on standard benchmarks. 2 In some applications there is focus on speed of training and on reducing computational costs -there it does make sense to focus on the performance of the typical model as opposed to the best out of n, so the use of mean or median is appropriate. the higher tier of the model performance population, and the use of mean may be inappropriate.Hence, rather than to completely abandon reporting the performance of the best model, we propose a way to fix its flaws. We do this by estimating the expected best-out-of-n (Boo n ) performance by running more than n experiments, which gives the estimate statistical validity if a sufficient number of experiments are run. We discuss how this measure relates to the performance distribution of the model, and we also give a method to empirically estimate Boo n .The paper proceeds as follows: First, we give a high-level explanation of why reporting performance of the best single model is problematic. We also give some evidence that it is widely used in the deep learning community , which is why this explanation may be needed. We proceed by presenting Boo n as a way to fix the above problems. We then give some experimental evidence for the flaws of best-singlemodel reporting and show that Boo n does not suffer from them. We wrap up by discussing the place of Boo n in a ML researcher's toolbox alongside traditional measures such as mean or median. Boo n does fix the main flaws of reporting the best single model performance. However, let us have a look at some of its limitations.Hyperparameter tuning This work does not fully compensate for improved expected results due to hyperparameter tuning, nor was it its primary aim. Boo n is appropriate in the case of random hyperparameter sampling, where the performances in different runs are independent. However, this is not the case for more advanced hyperparameter optimization methods. The primary focus of this work was on tackling variability due to random initialization, data shuffling, and similar sources, which we have shown to be significant in itself. Compensation for more advanced hyperparameter tuning (and ensuring the comparability of models in that case) is certainly a worthwhile area for future research.Mean, median, and other alternatives We do not claim our method to be strictly superior to traditional ways of aggregating results, such as mean or quantiles. However, we have outlined a case where using Boo n is justified -situations where a final model to be deployed can be chosen from a pool of trained candidates. In such case, Boo n is easily interpretable and more informative than a performance of a typical model, expressed by mean or median. Hence, we think Boo n is a useful addition to the methodological toolbox along existing methods.Just a single number Boo n is still just a single number whose ability to characterize the performance distribution is limited by its single dimension. Paper authors should try to characterise the performance distribution as fully as possible, which may involve a histogram, mean, standard deviation, ideally along a dataset containing the results of all experiments, from which an interested reader may be able to deduce whichever characteristic she finds interesting. Unfortunately, such characterization is usually lacking.However, alongside this detailed characterization, describing an architecture's performance by a single number still has its appeal, especially for the purpose of comparison among architectures and choosing the best one according to some criterion (in fact, each quantitative score can be understood as a proxy for ordering architectures with respect to such criterion of interest, such as expected performance of the best model out of n). We have explained why, in some cases, Boo n may be useful for such purpose.Computational cost Some may deem Boo n impractical due to its requirement to train architectures many times, which may be very expensive in some cases. However, result stochasticity needs to be addressed to produce reliable results, and it is hard to imagine a general method to do so without repeated evaluation 12 . Researchers should focus on architectures which they can evaluate properly given their resources. However, the main target of our criticism is not projects whose resources are stretched by a single training; it is projects that do have the necessary resources for multiple evaluations but use them to produce better-looking results rather than results that are more informative and robust. Reporting just the best single model performance is not statistically sound. This practice in machine learning research needs to change if the research is to have lasting value. Reviewers can play an important role in bringing this change.Still, asking for the performance of a best model out of n can have valid reasons. For the situations where the best-model performance is indeed a good metric, we are suggesting Boo n as a way to evaluate it properly. DISPLAYFORM0 where Φ is the c.d.f. of a standard normal random variable. DISPLAYFORM1 (the first integrand has the form of the p.d.f. found above and hence integrates to one) so the expected maximum is neatly expressed in terms of a maximum of a standard normal and is linearly proportional to both the mean and the standard deviation. Once n is fixed for comparison purposes, Boo n (N (0, 1)) is just a constant, e.g. Boo 5 FIG2 ) ≈ 1.163, Boo 10 (N (0, 1)) ≈ 1.539.
We point out important problems with the common practice of using the best single model performance for comparing deep learning architectures, and we propose a method that corrects these flaws.
{"src": "scitldr-AIC_train", "paper_id": "Skx5txzb0W"}