There is a growing movement to encourage reproducibility and transparency practices in the scientific community, including public access to raw data and protocols, the conduct of replication studies, systematic integration of evidence in systematic reviews, and the documentation of funding and potential conflicts of interest. In this survey, we assessed the current status of reproducibility and transparency addressing these indicators in a random sample of 441 biomedical journal articles published in 2000–2014. Only one study provided a full protocol and none made all raw data directly available. Replication studies were rare (n = 4), and only 16 studies had their data included in a subsequent systematic review or meta-analysis. The majority of studies did not mention anything about funding or conflicts of interest. The percentage of articles with no statement of conflict decreased substantially between 2000 and 2014 (94. 4% in 2000 to 34. 6% in 2014); the percentage of articles reporting statements of conflicts (0% in 2000,15. 4% in 2014) or no conflicts (5. 6% in 2000,50. 0% in 2014) increased. Articles published in journals in the clinical medicine category versus other fields were almost twice as likely to not include any information on funding and to have private funding. This study provides baseline data to compare future progress in improving these indicators in the scientific literature.
The inability to replicate published research has been an ongoing concern in the scientific community [1]. There is clear evidence from basic molecular and animal modeling research that a large portion of published articles lack reproducibility [2], which could potentially be related to the increase in lack of efficacy in clinical trials [3,4]. It has been suggested that the inability to replicate findings is due to a lack of research transparency [5]. Recently, there has been a growing movement to encourage making protocols, analytical codes, and data openly available [6–8]. In this study, we aimed to assess the current status of reproducibility and transparency in a random sample of published biomedical journal articles and to derive empirical data on indicators that have been proposed as being important to monitor in this regard [9], i. e. , the proportion of studies sharing protocols and raw data, undergoing rigorous independent replication and reproducibility checks, and reporting conflicts of interest and sources of public and/or private funding.
A total of 441 (88. 2%), from the original randomly selected 500 articles were publications in eligible research fields directly related to biomedicine. Of these, two-thirds had some form of empirical data (n = 304 (68. 9%) —n = 268 excluding case studies and case series, in which protocols and raw data sharing may not be pertinent, and n = 259 excluding also systematic reviews, meta-analyses and cost-effectiveness analyses where replication in studies with different data would not be pertinent). Among the 441 eligible studies, four (0. 9%) were cost effectiveness or decision analyses, 36 (8. 2%) were case studies or case series, 15 (3. 4%) were randomized clinical trials, five (1. 1%) were systematic reviews or meta-analyses, and 244 (55. 3%) were other articles with empirical data (including cross-sectional, case-control, cohort, and various other uncontrolled human or animal studies). Just over 30% of the articles were classified as research without empirical data or models/modeling studies. Less than one in five articles (19. 2%) had open full-text access from PubMed Central and about half (47. 8%) of the papers belonged to the journal category of clinical medicine (Table 1). Excluding case studies or case series (in which a protocol would not be relevant) 267 (99. 6%) of the 268 papers with empirical data did not include a link to a full study protocol. Only one article had a protocol; in fact, the article was itself the protocol of a trial, and it was published in the open-access journal Trials (A221). Another five studies either referenced their clinical trials identifier and included a link to ClinicalTrials. gov (A281, A434), provided only Clinical Trials identifiers (A407, A477), or stated that a Clinical Trials repository link was available on the journal website (A261), but none of these articles or their links contained information about a full protocol. There were seven other articles that had additional methods sections, figures, brief analytical plans and/or considerations, or supplementary materials either as a detailed appendix at the end of the paper (A434) or online (A25, A35, A174, A376, A290, A361 [contained an error message for page not found]). However, none of these supplementary materials fit our pre-specified definition of publicly available full or partial protocols. Of the 268 articles with empirical data (excluding case studies and case series) none provided access to all the raw data involved in the study. One article contained information on how to request a complete dataset (A287), two papers listed a non-functioning online link for supplementary data, data elements, or findings (A330, A361), and another four articles had supplementary files or links to some absorption spectra figures and/or data (A35, A117, A130, A305), but not to the entire raw data used in the paper. About half (51. 7%) of the 441 biomedical articles did not include any information on funding and about a third (34. 7%) were publically funded either alone or in combination with other funding sources. Of the 153 publically funded articles, 62 had National Institutes of Health (NIH) funding and four received National Science Foundation (NSF) support, alone or in combination with other sources of funding (Fig 1). There was no major change in the pattern of sources of funding over the 15-y period (S5 Fig). Of the 259 biomedical articles with empirical data, excluding case studies and case series, systematic reviews/meta-analyses, and cost effectiveness/decision analysis studies, only four (1. 5%) clearly claimed or were inferred to be replication efforts trying to validate previous knowledge. Over half (51. 7%) of the studies claimed to present some novel findings and four (1. 5%) had clear statements of both study novelty and some form of replication. There were 117 (45. 2%) articles that either had no statement or an unclear statement in the abstract and introduction about whether there were any novel findings or replication efforts. For the 259 biomedical articles with empirical data, excluding case studies and case series, systematic reviews/meta-analyses, and cost effectiveness/decision analysis studies, we also assessed whether any subsequently published papers had cited the article, mentioning that the authors were attempting to replicate part or all of their findings. Eight articles (3. 1%) in the final dataset had at least some portion of their findings replicated (A11, A59, A129, A222, A278, A285, A407, A441), while the remaining 251 articles had no citing article that claimed to be a replication. Five of these eight articles were from the clinical medicine journal category. Of the replicating articles, one was unable to reproduce the results from the original article (A59), but mentioned that different definitions were used [10]. Three articles had their results replicated through different methodology [11] (A222), [12] (A278), [13] (A407). One article (A11) had several subsequent studies that either confirmed portions of the original study [14] or failed to validate certain previous findings [15]. Two studies developed new methodology that the replicating studies confirmed (A285), either by comparing to available methods [16] or a newly developed method [17] (A441). One article (A129) was cited by a subsequent study by the same first author [18] that stated that one of their aims was to test a hypothesis from earlier observations with longer observation and modeling techniques. In order to measure whether empirical studies are eventually integrated in systematic reviews, the 259 articles with empirical data (again excluding case studies and case series, systematic reviews/meta-analyses, and cost effectiveness/decision analysis studies) were assessed on whether they had been cited at least once in subsequent systematic reviews and/or meta-analyses. Empirical data from 16 articles (6. 2%) were utilized in a systematic review/meta-analysis (A89, A93, A105, A157, A190, A222, A261, A268, A270, A278, A338, A340, A374, A407, A421, A477). At least one systematic review/meta-analysis cited another three articles but provided reasons for not including any of their data in a quantitative synthesis for any outcome (A83, A129, A221). Yet another 19 articles were cited incidentally by systematic reviews/meta-analyses (e. g. , in introduction or discussion, but without having data considered in quantitative syntheses for any outcome) (A5, A28, A31, A112, A203, A207, A224, A256, A274, A319, A322, A327, A377, A400, A413, A433, A435, A453, A463). Lastly, there were 221 articles (85. 3%) that were not cited in any systematic reviews/meta-analyses. The large majority of the 441 articles had no conflict of interest statement (305 [69. 2%]). Of the remaining, 110 (24. 9%) did not report any conflicts of interest and 26 (5. 9%) reported conflicts of interest. For the 15 randomized controlled trials, eight articles (53. 3%) reported no conflicts of interest, four (26. 7%) articles had no statement of conflict, and three (20. 0%) articles had a clear statement of conflict. Between 2000 and 2014, the percentage of articles with no statement of conflict decreased substantially (94. 4% in 2000 to 34. 6% in 2014), whereas the number of articles reporting statements of conflicts (0% in 2000,15. 4% in 2014) or no conflicts (5. 6% in 2000,50. 0% in 2014) increased (Fig 2). A comparison of articles published in journals in the clinical medicine category versus other fields showed some distinctive patterns (Table 2). Articles in the clinical medicine journal category were almost twice as likely to not include any information on funding and to have private funding, while they were far less likely to have public funding or funding from different types of sources (public and/or private and/or other). Articles in the clinical medicine journal category were also more likely to contain no statement on novelty or replication and less likely to claim novel study findings than articles in the “other” journal category. Furthermore, articles in the clinical medicine journal category were less likely to have full open access compared to other fields of study. There were no significant differences between replication, article citation for systematic review and/or meta-analysis, and statements of conflict (Table 2). When further limited to the 304 articles with empirical data, articles in the clinical medicine category journals were more likely to not mention funding (59. 0% versus 26. 3%) and less likely to have a PubMed Central reference number (PMCID) (16. 0% versus 27. 5%). There was no significant difference in the proportion of articles not including statements of conflicts of interest (65. 3% [clinical medicine] versus 71. 3% [other]).
Our evaluation is limited to published biomedical research information. In theory, sometimes one may be able to obtain additional raw data and protocols, and clarifications on conflicts or funding by communicating with the authors or sponsors. However, the yield would be uncertain, and personal communications should not replace the lack of transparency in the published scientific record. Furthermore, the fact that we only used the published records means that we could not correct any inaccuracies in the claims of the original authors. This may be particularly prominent in the case of claims for novelty, in which some authors may have tried to sell their paper as being more novel than it really is, so as to make it more attractive for publication. Although the two investigators (SAI and JDW) used their best judgment and discussed all eligible papers before agreeing upon a final classification, certain decisions may have been subjective. In particular, when determining study novelty and replication for articles from diverse biomedical fields, difficulty arose assessing whether study results were truly groundbreaking or being fully replicated. In order to account for these limitations, all ambiguous articles were discussed with a third reviewer (JPAI). We hope that our survey will further sensitize scientists, funders, journals, and other stakeholders in science to the need to improve these indicators. There are several efforts to improve reproducibility [40–42]. By continuing to monitor these indicators in the future, it is possible to track any evidence of improvement in the design, conduct, analysis, funding, and independence of biomedical research over time.
A sample of 500 English-language journal articles published between 2000 and 2014 was chosen randomly based on PubMed identification (PMID) numbers. PMID numbers ranging from 10,000,000 to 25,000,000 were inputted into OpenEpi (version 3. 02) random number generator to select a random sample of 750 PMID numbers. Beginning from the first number generated, each number was verified for eligibility in sequence until 500 eligible PMID numbers were chosen. Of the original 750 numbers, 742 were checked, with 242 being ineligible (54 did not have an article assigned, 100 were from before the year 2000,35 were not in English, and 53 were not in English and before the year 2000). The selected article distribution of PMID numbers (by year) was compared to the overall distribution of PMID numbers by year for English articles. The sample was found to be representative of the overall distribution (χ2 (df = 14), p > 0. 05). This sample size was chosen because given 500 articles and assuming that about half of them might have empirical data, if no article is found to fulfill the criterion for a transparency indicator, then the 95% confidence interval around that 0% estimate does not exceed 1%. Two investigators independently characterized and then cross-compared all extractions in groups of 50 articles at a time. Any uncertainties were first discussed in detail, and a third reviewer (JPAI) reassessed articles with arbitration discrepancies. The sample was characterized into seven study categories: (1) no research (items with no data such as editorials, commentaries, news, comments and non-systematic expert reviews), (2) models/modeling or software or script or methods without empirical data (other than simulations), (3) case report or series (humans only, with or without review of the literature), (4) randomized clinical trials (humans only), (5) systematic reviews and/or meta-analyses (humans only), (6) cost effectiveness or decision analysis (humans only), and (7) other (empirical data that includes uncontrolled study [human], controlled non-randomized study [human], or basic science studies). InCites Essential Science Indicators (ESI) was used to determine the main scientific field of each article. The journal for each index paper was searched in ESI in order to find the scientific field to which its Highly Cited Papers are ascribed. If a journal had articles ascribed to more than one scientific field, we examined the first five cited journals referenced by the index article. The journal names for these articles were then searched in ESI. If the majority belonged to the same field, this field was used for the index paper. If there was no majority, a field was selected based on the best judgment of the reviewers (JPAI, SAI, and JDW). If a specific journal was not found on ESI, we searched Journal Citation Reports (JCR) and identified the scientific field to which the highest-cited journal in the same JCR category had been ascribed to in ESI. Publications in scientific fields not directly related to biomedical research (chemistry, physics, computer science, economics and business, engineering, geosciences, material science, mathematics, physics, and space science) were further excluded from analysis. Even though these fields may sometimes have repercussions for biomedicine, their transparency practices may differ systematically, and their evaluation would require a separate, focused effort. Thus, 59/500 articles were excluded. JCR was used to determine 2013 journal impact factor. No information was recorded for journals without an impact factor for 2013. Availability of free access in PubMed Central was based on assignment of a PCMID (yes/no).
There is increasing interest in the scientific community about whether published research is transparent and reproducible. Lack of replication and non-transparency decreases the value of research. Several biomedical journals have started to encourage or require authors to submit detailed protocols, full datasets, and disclose information on funding and potential conflicts of interest. In this study, we investigate the reproducibility and transparency practices across the full spectrum of published biomedical literature from 2000–2014. We identify an ongoing lack of access to full datasets and detailed protocols for both clinical and non-clinical biomedical investigation. We also map the availability of information on funding and conflicts of interest in this literature. The results from this study provide baseline data to compare future progress in improving these indicators in the scientific literature. We believe that this information may be essential to sensitize stakeholders in science about the need for improving reproducibility and transparency practices.
Reproducible Research Practices and Transparency across the Biomedical Literature