Sources of Irreproducibility in Machine Learning: A Review

Lately, several benchmark studies have shown that the state of the art in some of the sub-fields of machine learning actually has not progressed despite progress being reported in the literature. The lack of progress is partly caused by the irreproducibility of many model comparison studies. Model comparison studies are conducted that do not control for many known sources of irreproducibility. This leads to results that cannot be verified by third parties. Our objective is to provide an overview of the sources of irreproducibility that are reported in the literature. We review the literature to provide an overview and a taxonomy in addition to a discussion on the identified sources of irreproducibility. Finally, we identify three lines of further inquiry.


Introduction
Progress in machine learning is to a large degree driven by empirical evidence. Model comparison experiments are the standard method to identify the best performing machine learning model for a given task [Melis et al., 2018;Sculley et al., 2018;Bouthillier et al., 2019;Dacrema et al., 2021], and clear wins are typically required for a research paper to be published at a top venue [Sculley et al., 2018]. Wagstaff [2012] proposed to focus research on real world impact challenges such as improve ELO rating of chess engines and discovering a new physical law using machine learning. Although several such impact challenges have been achieved lately, such as in game play, protein folding and software programming, model comparison studies are still the standard. Lately, however, several model comparison studies that benchmark state of the art results have shown that progress is not as steady as one could get the impression of when reading the scientific literature. Topics covered by such studies include forecasting [Makridakis et al., 2018], natural language processing [Belz et al., 2021a], generative adversarial networks [Lucic et al., 2018], deep reinforcement learning [Henderson et al., 2018], recommender systems [Dacrema et al., 2019], and image recognition [Bouthillier et al., 2019]. The * Contact Author findings establish that many state of the art results published at top venues in machine learning are not reproducible.
Proper methodology requires a good understanding of what can cause irreproducible results. An experiment conducted by Pham et al. [2020] where 16 identical training runs of a deep learning model resulted in test accuracy varying from 8.9% to 99%. According to a survey also done by Pham et al. [2020], 83.8% out of 901 participants were unsure or unaware about variance caused by how an experiment is implemented while most, although not all, participants knew about the algorithmic-level influences on variation in performance. This poor understanding of what can cause conclusion to be false in machine learning method development trickles down to the application areas that are downstream and use the methods developed by machine learning researchers, such as medicine [Roberts et al., 2021].
In the application areas, the stakes can be high and lead to unfortunate situations if relying on unsound evidence. While false findings and irreproducible results are expected products of the scientific method, the method is self-correcting if done correctly. However, methodological care has to be taken to reach sound conclusions when interpreting machine learning results. All too often, the conclusions reached are applied to a scope beyond what the original experiment was poised to suggest. While there are challenges regarding reproducibility that appear to be unique to the field of machine learning compared to other disciplines [Pineau et al., 2021], we do not focus on these alone in this paper. We review the literature to identify a larger scope of sources of irreproducibility and not only those that lead to a variation in results. In addition, we provide a taxonomy and a discussion on these sources of irreproducibility.

Reproducibility
Reproducibility is defined as the ability of independent investigators to draw the same conclusions from an experiment by following the documentation shared by the original investigators [Gundersen, 2021b], and a reproducibility experiment is the experiment conducted by independent researchers to confirm research findings. The documentation of a machine learning experiment is not restricted to written text in the form of a scientific report, it could also include the code and data used to conduct the experiment, but these are not required for independent investigators to conduct a reproducibility exper-  Figure 1: The scientific method is a systematic process for acquiring knowledge about the world: First, the world is observed (1) to form beliefs about it. A scientific theory is developed (2) to explain the causes and effects of the observed beliefs. A hypothesis is formed (3) in order to test the scientific theory. An experiment is designed and the research protocol is documented (4) to test the hypothesis. The experiment is carried out using the research protocol (5) and outcomes are produced (6). The experiment outcomes are analyzed (7) and the outcomes analysis is interpreted (8) with respect to the hypothesis. Finally, the beliefs of the world are updated (9) based on the results of the analysis and the scientific method is repeated by observing the world (10) with the updated beliefs.
iment. Still, code and data are commonly acknowledged as important for third parties to reproduce results [Haibe-Kains et al., 2020]. Having access to code and data reduces the effort required to reproduce the results, while also leading to increased trust in the research [Gundersen, 2019]. Drummond [2009] argues that the power of a reproducibility experiment is greater with increased difference from the original study, which can be enforced by providing less documentation.
There are three degrees of reproducibility that are derived from the steps of the scientific method (see figure 1). Outcome reproducibility is achieved if an independent researcher can produce the same outcome as the original experiment. If the experiment outcomes are reproduced exactly and the analysis and interpretation of the original investigators are sound, the results support the original hypothesis. Analysis reproducibility is when the outcomes by an independent investigator are different from the original investigators, but as long as the analysis of the outcomes as done by the original investigators lead to the same interpretation, the reproducibility experiment supports the original hypothesis. Interpretation reproducibility is achieved when a different analysis is done by independent investigators (on the same or different outcome), but the independent researcher's interpretation of the analysis supports the original hypothesis. If the interpretation of an independent researcher when applying the same analysis on the same or similar outcome does not support the original interpretation, the reproducibility experiment is inconclusive; new experiments are needed to confirm the findings.

Sources of Irreproducibility
The sources of irreproducibility are properties of the experiment that have not been properly controlled for by the original investigators. Sometimes even a small variation in these properties could change the interpretation of the analysis so that the belief update from the reproducibility experiment differ from the original study. As such, these properties could lead independent investigators to draw different conclusions after having conducted a reproducibility experiment. The sources of irreproducibility can be as diverse as the stochasticity of the machine learning algorithms, a bug in the performance metric code and the choice of baseline. The stochasticity of the algorithm can produce a highly unlikely outcome, so drawing any conclusions from it would be ill-advised. A bug in the performance metric will invalidate the analysis while a poor choice of baseline i.e. one that is actually not state of the art will invalidate the interpretation.
The sources of irreproducibility is a broader term than the sources of variation introduced by Pahm et al. [2020], which include algorithm-and implementation-level factors. We have identified a total of six groups of factors including those introduced by Pham et al.: 1) study design factors, 2) algorithmic factors, 3) implementation factors, 4) observation factors 5) evaluation factors, and 6) documentation factors. The factors are presented in the following sections 1 .

Study Design Factors
Study design factors are related to the methodology of an experiment and whether a study answers the research questions that are posed in tests suited for genuine evaluation.
Choice of baselines For many machine learning tasks, it is often not clear what comprises the state of the art. Studies have found that many deep learning papers only compared against other deep learning algorithms, even though they were not performing better than other simpler baselines [Cremonesi and Jannach, 2021]. This could happen in cases where progress is shown by reusing experimental designs that propagate weak baselines without questioning them [Dacrema et al., 2021].
p-fishing is when seeking statistically significant results beyond the original hypothesis [Cockburn et al., 2020].
p-hacking Cockburn et al. [2020] use the term collectively for research practices that allow manipulation of experimental and analysis methods to produce statistically significant results.
HARKing (Hypotheses After Results are Known) is posthoc reframing of experimental intentions to present a pfished outcome as having been predicted from the start. Study design factors lead to poorer outcomes for both baselines and false interpretations of the results as the basis of the interpreations is unsuited experiment designs and poor experiment setup. The factors are controlled through ensuring that all models are given the same amount of computation budget and validation, that is finding out whether the right study is done. Validation procedures could be improved through registered reports and pre-registration [Cockburn et al., 2020], which there are few if any examples of in artificial intelligence and machine learning research [Gundersen, 2021a].

Algorithmic Factors
Algorithmic factors are properties of the learning algorithms and training process that might affect the performance of an algorithm. Batch ordering is when the batch order is randomized for every epoch [Pham et al., 2020;Bouthillier et al., 2021].
Algorithmic factors describe stochastic properties of machine learning methods and they affect the outcome of an experiment. Hence, the variation in outcome over several experiment runs must be reported. As pointed out by Miller and Miller [2018]: "No quantitative results are of any value unless they are accompanied by some estimate of the errors inherent in them." Algorithmic factors can be controlled by setting the pseudo-random number generator initialization seed so that the outcome will be the exact same for each experiment run, given that all other things are equal.

Implementation Factors
Implementation factors are related to which laboratory conducts the experiment, as they include the hardware and ancillary software used to conduct the experiment. Non-deterministic ordering of floating-point operations Truncation error of floating point calculations introduces variability as A + B + C = C + B + A, which is a problem when such operations are calculated in parallel, as they are for GPUs [Pham et al., 2020;Gundersen et al., 2022]. Truncation error can be reduced by changing from single precision (32 bits) to double precision (64 bits) at a cost of doubling memory requirements and tripling the training time [Pinto et al., 2021].
Parallel execution Random completion order of parallel tasks introduce variation [Pham et al., 2020]. Increased parallelism is a driver for implementation noise [Zhuang et al., 2021]. Compiler settings Hong et al, [2013] tested sensitivity to Intel compiler optimization levels for experiments that relies on huge amounts of floating point calculation, such as weather simulations. Auto-selection of primitive operations High level libraries implement deep learning algorithms using GPUoptimized deep learning primitives provided by lowlevel libraries such as cuDNN and CUDA [Pham et al., 2020]. Autotune in cuDNN automatically benchmarks several modes of operation for primitive functions in run-time, which might change between runs. Processing unit Changing the processor can affect results [Hong et al., 2013]. Nagarajan et al. [2019] found that a deterministic GPU implementation repeatedly generated the same result when executed on the same GPU, but changed to a different, but deterministic result when being executed on another GPU.
Implementation factors cannot be fully controlled unless the experiments are repeated on the same computer, in which the results are not reproduced, but repeated. However, they can be controlled partly by not using multiple processes that cannot guarantee data order, deterministic implementations of primitive operations, use single thread process and force running in serial manner [Pham et al., 2020], and conducting the experiment in a Docker container with the same seed. Difference in results caused by floating point precision, conducting the experiments on a HPC, and differing processing units cannot yet be controlled for.

Observation Factors
Observational factors are related to the observations that are made as part of the reproducibility experiment, which for most machine learning experiments are offline data from measurements and sensors, but also include online data and benchmark gyms used in reinforcement learning. Dataset bias The methods used to gather data (manual or automated) and the way data is captured (objects are often centered when photographed) bias datasets [Torralba and Efros, 2011]. Recth et al. [2019] show that algorithms generalize poorly even on datasets that are replicated using the same source populations, so the lack of access to data used in the original study could lead to differences in data distribution for reproducibility experiments [Pineau et al., 2021]. In the social and environmental sciences, models might not generalize from one geographic area to another because of spatial dependence and heterogeneity [Goodchild and Li, 2021]. Pre-processing Differences in data pre-processing could change outcomes [Dacrema et al., 2021;Gundersen, 2021b]. Data augmentation Stochastic data augmentation procedures will affect how data is augmented, and difference the resulting difference in training data will lead to different performance [Pham et al., 2020;Zhuang et al., 2021;Bouthillier et al., 2021]. Data splits Difference in data splits cause difference in results [Makridakis et al., 2018], which includes stochastic sampling from the training set instead of training on a static validation set [Bouthillier et al., 2019;Bouthillier et al., 2021]. Also, random selection of samples during training is typically used in gradient boosted trees [Pouchard et al., 2020]. According to Gundersen and Kjensmo [2018], only 16% specify the validation set and 30% specify the test set, which means that outcome reproducibility is not possible to achieve. A 2% to 12% variation has been shown in labeled attachment scores of Natural Language Processing experiments when comparing models trained using standard test splits versus random test splits, which suggests that the results reported by experiments that only use the standard test splits can be influenced by a bias in the standard test splits that favor certain types of parsers [Çöltekin, 2020]. Environment properties Stochasticity and different dynamic properties of the testing environment could affect the variability of the performance, especially in continuous control domains such as those used by deep reinforcement learning [Henderson et al., 2018]. How data was gathered and prepared will affect the outcome of the experiment. Observation factors can be partly controlled by setting the random seed, sharing details about preprocessing and data provenance, that is keeping a record of how the data set was created [Gebru et al., 2021]. What is done with duplicate data, outliers and missing values can introduce bias into the dataset [Stodden, 2015]. As long as datasets are finite samples of an infinite population [Melis et al., 2018], they cannot be fully controlled.

Evaluation factors
Evaluation factors are related to how the investigators reach the conclusions from doing an experiment.
Selective Statistical analysis Improper use of statistics to analyze results, such as claiming significance without proper statistical testing or using the wrong statistic test lead to false conclusions [Card et al., 2020;Pineau et al., 2021]. Power analyses should be done prior to evaluation when comparing against baselines, and the number of instances in the test will determine the effect size and should be chosen accordingly [Card et al., 2020].
Test data issues Data leakage results in overestimating the performance as a model is trained on data that should only be available at test time [Dacrema et al., 2021]. Götz-Han et al. [2020] demonstrate five cases of reported performance gains well above the state of the art that were the results of data leakage. The performance results far below the claims of the original researchers when the data leakage errors were corrected. Cases where metrics have been reported on training data instead of test data has been found . Evaluation factors could affect both the outcome and the analysis and thus lead to the wrong interpretation. As documentation factors, evaluation factors can only be controlled through validation and ensuring that one is doing the right experiment. Sculley et al. [2018] list the following practices that should be included in empirical studies: 1) tuning methodology, 2) sliced analysis, 3) ablation studies, 4) sanity checks and counterfactuals and 5) at least one negative result. However, they noted that we have not seen a material increase in standards for empirical analysis or rigor across the field.

Documentation Factors
Documentation factors are related to whether the original investigators share enough information about their experiment for third parties to reproduce the results. This means documenting the factors listed above.Additional factors include: Readability The readability of papers influences whether it is possible to reproduce results. Mathiness could lead to reduced readability [Lipton and Steinhardt, 2019]. Gundersen and Kjensmo [2018] found that a large degree of papers only implicitly state what research questions they answer (94%), which problems that they seek to solve (53%), and what the objective (goal) of conducting the research is (78%). Only 5% of the papers explicitly states the hypothesis and 54% contain pseudocode. Raff [2019] found that number of tables, readability, specification of hyperparameters, pseudo-code, number of equations and compute needed to run the experiment correlated strongly with reproducibility. Experiment design details Under-specification of the metrics used to report results and misspecification or underspecification of the model or training procedure might lead to the wrong conclusions [Pineau et al., 2021]. Workflow The exact steps taken and their order when conducting the empirical machine learning studies will affect the outcome [Gundersen, 2021b], especially when they become more complex [Rupprecht et al., 2020]. Missing documentation of steps could also affect the reproducibility.  found that whether data was augmented was not specified, even when it was. . Documentation factors affect all reproducibility degrees, but can mostly be controlled by sharing code and data. This is why code and data sharing is so effective for increasing reproducibility. Documentation factors will support validation of study design, the evaluation and observation factors as well as verification of an experiment.

Discussion
This review has shown that there are many sources of irreproducibility, and there are probably others that we have not covered. Because of this, it cannot be expected that all experiments published as scientific articles will be able to control for all the sources of irreproducibility covered in this review. Controlling for all these sources is time-consuming, requires a huge effort in configuration, requires too many computational resources as well as more diversity between the resources than can be expected from a single lab. The value of the taxonomy presented here is as a resource for helping the machine learning community to better understand the possible sources of irreproducibility that could affect the results of machine learning experiments and render the conclusions made from them invalid. As such, the taxonomy could help inform the limitations of experiments, so that over-claiming of results can be avoided to a greater extent.
One way of managing expectations of which sources of irreproducibility should be controlled in a study is to distinguish between confirmatory and hypothesis generating studies. Confirmatory studies test explicit and precise hypotheses about the effect of factors while in hypothesis generating studies hypotheses are suggested and help design experiments [Cohen, 1995]. Expectations about reproducibility would be lower for hypotheses generating studies while they would be higher for confirmatory studies. We propose that study design and observation factors can be relaxed while implementation and evaluation factors do not have to be given much consideration for hypothesis generating studies. Only algorithmic (when relevant) and documentation factors should be given attention for hypothesis generation studies. For confirmatory studies, one could argue that all relevant factors should at least be discussed if not controlled for.
Another way of managing expectations of which sources of irreproducibility to be controlled is to distinguishes between inter-and intra-laboratory reproducibility. This is done in other domains of scientific discovery, such as analytical chemistry [Miller and Miller, 2018]. Inter-laboratory reproducibility is a measure of variance in results between laboratories, while intra-laboratory reproducibility measures variance in the results within the laboratory. A similar distinction can be made in machine learning if one considers differing hardware and ancillary software as inter-laboratory conditions. This means that for experiments run in one laboratory intra-laboratory variation should be reported. Could studies that do not consider inter-laboartory reproducibility be considered confirmatory at all? Ioannidis [2005] suggests that care should be taken for small effect sizes, which is defined as less than 1.5%. Pham et al. [2020] found that implementation factors results in a 2.9% difference in performance when conducting the exact same experiment, so many state of the art results are likely just caused by implementation factors.
A striking finding that was commented on by Belz et al. [2021a] is that reproduction under same conditions far more frequently yields results that are worse than results that are better. However, it is non-trivial to determine the number of identical runs needed to decide which algorithm is best. A solution Pham et al [2020] propose is to iteratively performing more experiment runs where results are compared to a selected baseline; the process can stop when statistical tests confirm the significance of the difference between the algorithms.
This review shows that the interest in the methodological issues caused by the apparent stagnation of some sub-fields of machine learning has grown lately. Still, some challenges remain. We identify three main lines of inquiry: 1. Methodology: Methodologies must be developed that lay out how experiments can be designed to control for the sources of irreproducibility.
2. Control: Which sources of irreproducibility different experiment types are expected to control for should be identified. 3. Distributed scenarios: How will distributed data and computation affect the reproducibility of results, and how can the sources of irreproducibilty be controlled for in these scenarios? The community needs better guidance in how to improve reproducibility of experiments and need to better understand how distributed experiments will affect reproducibility. A proper understanding of the sources of irreproducibility is a first step.