The preclinical research process is a cycle of idea generation, experimentation, and reporting of results. The biomedical research community relies on the reproducibility of published discoveries to create new lines of research and to translate research findings into therapeutic applications. Since 2012, when scientists from Amgen reported that they were able to reproduce only 6 of 53 “landmark” preclinical studies, the biomedical research community began discussing the scale of the reproducibility problem and developing initiatives to address critical challenges. Global Biological Standards Institute (GBSI) released the “Case for Standards” in 2013, one of the first comprehensive reports to address the rising concern of irreproducible biomedical research. Further attention was drawn to issues that limit scientific self-correction, including reporting and publication bias, underpowered studies, lack of open access to methods and data, and lack of clearly defined standards and guidelines in areas such as reagent validation. To evaluate the progress made towards reproducibility since 2013, GBSI identified and examined initiatives designed to advance quality and reproducibility. Through this process, we identified key roles for funders, journals, researchers and other stakeholders and recommended actions for future progress. This paper describes our findings and conclusions. The biomedical research community relies on the reproducibility of published discoveries to create new lines of research and to translate research findings into therapeutic applications. Irreproducibility limits the translatability of basic and applied research to new scientific discoveries and applications.Preclinical biomedical research is the foundation of health care innovation. The preclinical research process is a cycle of idea generation, experimentation, and reporting of results 1. The bAlthough quality control during the research process centers on review of proposals and completed experiments , opportu2. Global Biological Standards Institute (GBSI) released the “Case for Standards” in 20131, one of the first comprehensive reports to address the rising concern of irreproducible biomedical research. Further attention was drawn to issues that limit scientific self-correction, including reporting and publication bias, underpowered studies, lack of open access to methods and data, and editorial and reviewer bias against publishing reproducibility studies (see Section IV)3. Based on these findings, GBSI completed an economic study in 2015 and estimated that the prevalence of irreproducible preclinical research exceeds 50%, with associated annual costs of approximately $28B in the United States alone4.The magnitude and effects of reproducibility problems are well documented. In 2012, scientists at Amgen reported that they were able to reproduce only 6 of 53 “landmark” preclinical studies6. The Report addresses progress in the four major components of the research process: study design and data analysis, reagents and reference materials, laboratory protocols, and reporting and review. Moreover, the Report identifies the following broad strategies as integral to the continued improvement of reproducibility in biomedical research: 1) drive quality and ensure greater accountability through strengthened journal and funder policies; 2) engage the research community in establishing community-accepted standards and guidelines in specific scientific areas; 3) create high quality online training and proficiency testing and make them widely accessible; 4) enhance open access to data and methodologies.Research community stakeholders have responded to these concerns with innovation and policy. In early 2016, GBSI launched the Reproducibility2020 Initiative to leverage the momentum generated by these stakeholder-led initiatives. Reproducibility2020 is a challenge to all stakeholders in the biomedical research community to improve the quality of preclinical biological research by the year 2020. The Reproducibility2020: Progress and Priorities Report (or Report), is the first to highlight progress and track important publications and actions, since the issue started to get broad research community and public attention in 2013Note to Reader: Terms such as reproducibility, replicability, and robustness lack consistent definition. The Report draws upon the definitions promulgated by the framework proposed by Goodmanet al.7: “methods reproducibility” refers to the complete and transparent reporting of information required for another researcher to repeat protocols and analytical methods; “results reproducibility” refers to independent attempts to produce the same result with the same protocols ; and “inferential reproducibility” refers to the ability to draw the same conclusions from experimental data. The Report defines “reproducibility” to include issues affecting any of these three areas.4 . Poor study design and incorrect data analysis can sabotage even a perfectly executed experiment.Study design is the development of a research framework and analytical methods prior to beginning experiments10. These findings can promote a multifaceted approach to improving study design and data analysis. Although researchers ultimately are responsible for ensuring sound study design and analysis, funder policies should encourage rigorous study design before research begins, journal requirements should facilitate better review of completed research, and training and support resources should improve researchers’ study design and analysis skills.Researcher surveys suggest that study design flaws are a key source of irreproducibility. Four of the top ten irreproducibility factors identified in a researcher survey relate to poor study design and analytical proceduresNIH study design policy. Funder policies that require good study design are especially powerful because they encourage researchers to develop rigorous study plans before beginning experimentation. Clinical research has regulatory mechanisms to review study design; for example, Phase 2 and 3 Investigational New Drug clinical trial applicants must acquire FDA approval of the study design and statistical analysis plan that includes explicit description of contingencies, such as sample exclusion criteria (http://www.accessdata.fda.gov/SCRIPTs/cdrh/cfdocs/cfCFR/CFRSearch.cfm?CFRPart=312). Preclinical biomedical research is not covered by these regulatory standards, and generally has not required explicit justifications of key parameters, such as sample sizes and statistical tests, in the hypothesis and specific aims sections of proposals or in publications. For example, an analysis of 48 neuroscience meta-analyses found that 28 (57%) of the studies had a median study power of 30% or less, despite the relative ease of increasing sample size11. The new NIH policy (seehttp://www.csr.nih.gov/CSRPRP/2016/09/implementing-new-rigor-and-transparency-policies-in-review-lessons-le). Formal evaluations of this ongoing effort will provide valuable lessons for NIH and other funders interested in implementing their own rigor and transparency guidelines.licy see requiresTo augment these efforts, NIH has worked with the journal community to develop publication guidelines (see Section IV), and funded the development of researcher training programs in study design (see “Training and Support” below) as part of its rigor and reproducibility efforts.9. NIH serves as an important model for other government and private research funders looking to establish greater accountability around quality and rigor.As the largest and most influential research funder in the world, NIH took a major step in establishing new guidelines and going on record that NIH will address other areas where they can impact reproducibilityNIH Rigor and Transparency Guidelineshttps://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-011.html,https://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-012.html) This policy includes applicant and reviewer guidance in four key areas: scientific premise, scientific rigor, consideration of sex and other biological variables, and authentication of key biological and/or chemical resourcesNIH’s Rigor and Transparency Guidelines went into effect on January 25, 2016  Applicants are required to describe the strengths and weaknesses of prior studies cited in their scientific premise, specifically they are required to describe and justify the proposed study design, and develop authentication plans based on established standards. Since reviewers are now instructed to review applications based on these criteria, grant applicants that fail to meet the new criteria are less likely to be funded. NIH also requires grantees to report on rigor and transparency measures in their publications and the Research Performance Progress Reports submitted during the life of an award. These new guidelines underscore the need for development and propagation of study design training, pre-registration resources, and low cost authentication tools. For further information, see the NIH webpage:https://grants.nih.gov/reproducibility/index.htm . As these principles do not specify what these requirements should be, implementation varies by journal. One example from theBiophysical Journal recommends that authors consult with a statistician and requires reporting of specific information about sample sizes and statistical analyses .In Vivo Experiments (ARRIVE) guidelines developed by the National Centre for the Replacement Refinement & Reduction of Animals in Research, include a checklist for researchers who perform animal studies to help researchers appropriately report study design and sample size justifications (www.nc3rs.org.uk/arrive-guidelines). These guidelines can also be used to help ensure that researchers are planning their animal experiments correctly. As of January 2017, these reporting guidelines have been endorsed by nearly 1,000 journals and are required by the major funders in the UK, including the Wellcome Trust and Medical Research Council .In the United Kingdom, the Animal Research: Reporting ofhttps://cos.io/rr). Once study design has been approved, participating journals essentially guarantee publication so long as the authors follow the study design. In addition, researchers can use the Registered Reports format to submit articles to these journals. Currently, 45 journals are participating in this initiative. In a separate, but related initiative, the Center for Open Science’s Pre-Registration Challenge has been designed to provide training and incentives for up to 1,000 researchers to pre-register study protocols and submit manuscripts to participating journals (https://cos.io/our-services/prereg/).Some journals are prototyping alternate review models to help verify study design. As of January 2017, the Registered Reports initiative through the Center for Open Science allows selected reviewers to comment on study design and methods prior to data collection . Statcheck and StatReviewer are tools developed by researchers to automatically review data analysis information contained in published manuscripts16. Researchers also have broadly deployed the Statcheck tool on thousands of published studies (see Section IV).One journal,Training and support. Many life-science researchers will require training and support to satisfy the funding and publication policies described above. In the 2016 Proficiency Index Assessment (PIA) 18 and journal articles19 can be used for course development or independent study by more senior trainees.PIA) see, GBSI su14.New approaches to training researchers should be a priority for all steps in the research cycle, including the study design training resources described in the Report. Enhanced training should be available for all levels of researchers—graduate students, post-docs, and experienced PIs. Active learning opportunities are particularly important, considering the informal apprenticeship culture of science, in which trainees learn how to design, perform, and report on their research by working with more senior scientists. However, not all senior researchers have the most current expertise or may not be able to spend the requisite time with their trainees. Surveys of researchers support this need: the 2016 Proficiency Index Assessment indicated that even experienced researchers stand to benefit from study design training, and a figshare and Digital Science survey reported that over half of researchers wanted training on open access policies and proceduresInnovative pedagogical approaches are required to ensure that training is effective and engaging for researchers at all stages of their careers. These approaches, including interactive teaching, in-lab practice, and proficiency assessments, are increasingly being explored by many institutions (see “Training and Support” example in Section I). Online training modules are a cost-effective way to provide high-quality, accessible, interactive training for researchers at all levels.20 and Harvard University (https://nanosandothercourses.hms.harvard.edu/node/96) demonstrate the value of study design training. These courses are becoming more widespread and better tailored to the needs of life scientists, but are not universally available or required. Efforts are underway to increase the experimental design skillset of early-career students, but funding in this area has been relatively modest and in general, private funders have seen training and education as the responsibility of government funders and graduate programs. In 2014, NIH funded graduate courses on study design. Since 2014, NIH has issued a series of four funding opportunities for grantees interested in providing study design instruction for their graduate students and postdoctoral trainees through administrative supplements to existing grants . Several of these grantees have used the funds to develop study design training programs that are tailored to their respective research areas (https://www.nigms.nih.gov/training/instpredoc/Pages/admin-supplements-prev.aspx). For more computationally-focused researchers, a Harvard course on reproducible genomics is available online for free21.The positive response to study design courses established at Johns Hopkins Universityhttp://statistics.berkeley.edu/consulting,https://catalyst.harvard.edu/services/biostatsconsult/,http://www.stat.purdue.edu/scs/), and the Center for Open Science provides a similar service (https://cos.io/our-services/training-services/). The CHDI Foundation provides protocol and study design assistance, evaluation, and review to researchers studying Huntington’s disease . This model may be of interest to other disease-specific funders as a low-cost investment that can improve research rigor and strengthen the community of practice in their mission area.In addition to training, researchers now have increased access to expert support during study design and analysis. University statistics departments often provide free consulting services to affiliated researchers , few standardized biological materials exist. However, surveys suggest that life science researchers increasingly understand the need for standardized materials1, and the research community recently has made progress on cell line authentication and antibody validation.Materials developed and validated based on standards are well-characterized and demonstrate consistency. Standardized materials that exhibit a predictable behavior can be used reliably in methods reproducibility, and can facilitate development of reference materials for assay validation. Standards of most well-known and often-used biological materials typically apply to particular clinical applications, such as virus strains used in influenza vaccine developmentStandards development for biomedical research reagents. Stakeholders of preclinical research include researchers, reagent manufacturers, funders, journals, standards experts, and nonprofit organizations from countries throughout the world. Recent efforts to establish antibody databases, information-sharing requirements, and international frameworks for antibody validation standards are good examples of the broad, multi-stakeholder approach required to develop consensus standards around a specific reagent . Antibodies are key reagents in preclinical research for activities as diverse as protein visualization, protein quantification, and biochemical signal disruption. Antibody performance is variable, with differences in specificity, reliability, and functionality for different types of experiments , manufacturers, and lots, harming reproducibility24. Stakeholder solutions include antibody databases, such as the CiteAB database (https://www.citeab.com/), and repositories, such as the proposed universal library recombinant antibodies for all human gene products25. In all cases, validation is a key component of the solution.The research community has acknowledged that antibodies are an area of widespread error and inaccuracyhttps://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-011.html) providing additional impetus for new standards, policies, and practices. Researchers, manufacturers, pharmaceutical companies, funders, and journals have held dedicated conferences on antibody validation e.g. . In 2016, the International Working Group on Antibody Validation (IWGAV) qualitatively identified key validation “pillars” that may be suitable for assessing antibody performance26. Seeking to build on the IWGAV recommendations, GBSI and The Antibody Society organized a workshop for all stakeholder groups to develop actionable recommendations to improve antibody validation27. Stakeholder groups recognized the shared responsibility of antibody validation and effective communication of validation methodology and results. In addition, they highlighted the need for continued, multi-sectoral engagement during the development of standards for validation, which may vary by use case, and information-sharing, which may vary by stakeholder.NIH specifically highlights antibody authentication in the Rigor and Transparency guidelines,  and many journals now ask researchers to perform cell line authentication .As with study design, cell culture validation can be enhanced with policies from funders and journals. For example, the Prostate Cancer Foundation has been a leader in validation of cell lines used to study the disease, requiring periodic cell line authentication since 2013. NIH now requires grant applicants to describe their authentication plan as part of the Rigor and Transparency guidelines (ANSI/ATCC ASN-0002-2011. The authentication assay uses Short Tandem Repeat (STR) profiling technology and is an affordable cell line authentication tool. The International Cell Line Authentication Committee’sDatabase of Cross-contaminated or Misidentified Cell Lines provides researchers with a dataset to check during the authentication process31. For products of animal origin, U.S. Department of Agriculture regulations specify testing protocols for mycoplasma and select viruses32 and test kits are commercially available.Many of the validation assays required for cell culture validation can be borrowed directly from other applications. In 2011 and 2012, ATCC organized an international group of scientists from academia, regulatory agencies, major cell repositories, government agencies, and industry to develop a standard that describes optimal cell line authentication practices,33. GBSI’s #authenticate campaign encourages this kind of stakeholder engagement (www.gbsi.org/authenticate).Improving the reproducibility and translation of biomedical research using cultured cell lines must build on ongoing, multi-stakeholder efforts to raise awareness of the issues of misidentification and the role of authenticationTechnology and assay development. The development and propagation of standards is an iterative process. For example, recent publications highlight the simultaneous progress in cell line authentication technologies and standards development, including the establishment of reference data standards and cell line authentication policies for the broader research community29. As technology development progresses, the standards need to be revisited and improved to reflect the current capabilities afforded by new tools34. For example, more affordable next generation sequencing is an increasingly useful tool to validate genome editing and characterize changes in cell behavior35, and mass spectrometry and lab-on-a-chip assays can help characterize sera and other liquid reagents37.Sera validation: an opportunity for standards and technology development. One opportunity to further improve cell culture validation would be to develop standards for sera production and validation. The media used to feed most cells in culture include sera, such as fetal bovine serum, that provides a variety of growth factors and other small molecules. Even authenticated cells may perform very differently in two different sera preparations. Serum is a “black box” ingredient with high variability between manufacturers and lots. Recently developed best practices include characterizing and reporting information on the particular lot(s) of serum/sera used in an experiment, and repeating an experiment with multiple lots of sera to ensure that observed phenotypes are not serum-related artifacts38. Serum manufacturers have begun to characterize and validate sera , but no industry standard exists for reporting serum characteristics and reliability.38. Researchers are developing and validating robust, serum-free culture systems. Clear material and validation standards are building blocks that facilitate this development.Further technological development could reduce reliance on sera. In serum-free culture, researchers precisely define all components of the cell culture medium rather than using a “black box” serum. Building a system with defined minimum essential components improves reproducibility and enhances scientific understanding of the key signaling molecules involved in biological processes of interest13. Despite this relative confidence in their laboratory execution skills, researchers frequently are unable to recreate an experiment based on the experimental methods published in journals, which usually do not contain step-by-step laboratory protocols that specify every relevant variable. Further, a particular study may use a modified version of an established protocol, but state the method was “as previously described” without noting the changes. If attempts to contact authors to request the original protocols are not successful, the reader may not be able to reproduce the methods in the published work. In aNature survey, nearly half of researchers felt that incomplete experimental protocol descriptions in published articles hindered methods reproduction efforts10. Although fewer efforts exist in this key area than in the other three areas described in this report, newly developed tools and processes designed to facilitate protocol sharing and version control may improve documentation and reduce barriers to methods reproduction.Reproducibility requires thorough, detailed laboratory protocols. Without ready access to the original protocols, researchers may introduce process variability when attempting to reproduce the protocol in their own laboratories. The respondents of the GBSI’s Proficiency Index Assessment were more confident in their experimental skills than their study design skillsProtocol repositories. Protocol repositories are an innovative approach that may facilitate transparency, protocol sharing, and version control. Researchers can upload their protocols to a repository, such asProtocols.io, precisely specifying all step-by-step instructions with links to required reagents. As the original researchers, or others, modify the protocol, they can document these changes in the repository and create their own “forked” version of the protocol. Protocols in the repository can receive a DOI number, making identification of the precise version used in a publication easier. Suppliers also can post recommended protocols for their products on these websites, which facilitates adoption of their products.https://www.moore.org/article-detail?newsUrlName=$8m-awarded-to-scientists-from-the-gordon-and-betty-moore-foundation-to-accelerate-development-of-experimental-model-systems-in-marine-microbial-ecology,https://www.protocols.io/groups/protist-research-to-optimize-tools-in-genetics-protg). As of January 2017, this group has 95 members who have contributed 31 protocols to the platform. Although this group does not focus on preclinical research, the practices established by this group are a relevant example that could be reproduced in preclinical research. Preclinical research funders may find added value with version control, protocol forking, and communities of practice in their areas of interest.Protocol development requires a robust community of practice, so that protocols can be developed and tested by researchers in different laboratories. This practice ensures that the written instructions are understandable and replicable by a third party. Emerging on-line tools, such as BioSpecimen Commons (The Biodesign Institute at Arizona State University), provides a common location and uniform set of protocols and conditions for clinical sample-related standard operating procedures. Another example is the international Protist Research to Optimize Tools in Genetics group, funded by the Gordon and Betty Moore Foundation, and working on the Protocols.io website  However, most methods sections still do not contain step-by-step protocols. Authors submitting to participating journals can include links to Protocols.io in the methods section, specifying the exact version of a protocol that was used in the study with a DOI number (https://www.protocols.io/partners?publishers). In April 2017, PLOS and Protocols.io announced a partnership where PLOS is encouraging their authors to log their experimental methods in Protocols.io (https://www.moore.org/article-detail?newsUrlName=open-access-to-data-and-the-laboratory-methods).Journal of Visualized Experiments, publish original, peer-reviewed manuscripts and videos of both established and new techniques (http://www.jove.com/). The use of videos helps to communicate technique subtleties that may not be captured in written instruction. This type of tacit knowledge often only can be obtained by visiting a laboratory and learning directly from the protocol developers.Although methods journals  usually provide sufficient information about protocols, most scientific publications do not. Even new techniques are not described in full detail because they build on established techniques, the methods for which are not fully described. However, some journals, such as theThe scientific community requires ready access to publications and the original underlying data to adequately review studies and conduct results for reproducibility efforts. Journal reporting guidelines improve methods reproducibility by ensuring that manuscripts contain a minimum standard of required information. Data standards further facilitate this process, as large data sets formatted in an agreed-upon, machine-readable format are easier to find, compare, and integrate across different studies. With better access to data and manuscripts, researchers now can engage in more robust post-publication review. Reducing these barriers can improve reproducibility by identifying potential flaws in published papers, making scientific self-correction and self-checking faster and cheaper.Enhanced journal reporting guidelines. Journals increasingly recognize the importance of methods reproducibility and are developing more transparent and enhanced reporting guidelines. Co-led by the Nature Publishing Group, the American Association for the Advancement of Science , and the NIH (as part of its Rigor and Reproducibility efforts), the scientific journal community established the Principles and Guidelines for Reporting Preclinical Research in June 2014 . Per the last update of the NIH website in 2016, 31 journals have signed on to these guidelines . The guidelines provide a minimum consensus standard for statistical rigor, reporting transparency, data and material availability, and other relevant best practices, but do not specify in detail exactly what these reporting requirements should be.39. Several of the journals highlighted in the examples listed below are signatories to the TOP guidelines.More specific guidelines from journals have built upon this initial effort. Differences in implementation of reporting guidelines may cause some short-term confusion among authors and reviewers. However, over time, their implementation could provide long-term benefit in identifying successful approaches and best practices. One initiative that seeks to provide broad direction and even instruction to journals are the Transparency and Openness Promotion (TOP) Guidelines, promulgated by the Center for Open Science’s Open Science Framework. TOP includes templates for journals interested in implementing their own reproducibility guidelines, and exist in a tiered framework so journals can gradually implement more stringent standards as they improve their own implementation and review capabilityBiophysical Journal are an example of what enhanced journal guidelines look like in practice. These guidelines specifically establish reporting standards in four key areas: Rigorous Statistical Analysis, Transparency and Reproducibility, Data and Image Processing, and Materials and Data Availability .Expanded reproducibility guidelines from thehttp://www.nature.com/authors/policies/checklist.pdf).Authors submitting to the Nature Publishing Group family of journals must complete a reporting checklist to ensure compliance with established guidelines, including a requirement that authors detail if and where they are sharing their data (http://www.cell.com/star-methods).STAR Methods guidelines  are designed to improve reporting across Cell Press journals. These guidelines remove length restrictions on methods, provide standardized sections and reporting standards for methods sections, and ensure that authors include adequate resource and contact information (http://www.hhmi.org/sites/default/files/About/Policies/sc_300.pdf).Since January 2016, researchers funded by the Howard Hughes Medical Institute have been required to adhere to a set of publication guidelines that cover similar areas as the minimum consensus guidelines described above  stakeholder feedback studies; and 2) research measuring the frequency of compliance over time. The journal community periodically should reconvene and use data from these evaluations to identify and propagate successful implementation of the Guidelines, and to update and improve the Guidelines.Open access policies. Funder policies increasingly mandate access to data and publications  expressed an interest to set up a “publishing platform” to stimulate open-access publishing in Europe42. The EC is hopeful the platform will catalyze their initial plan to make all published research funded by EU members open access by the year 2020 (http://www.sciencemag.org/news/2017/03/european-commission-considering-leap-open-access-publishing).ions see. As of Ohttps://www.ucl.ac.uk/library/open-access/research-funders] contains a summary of many institutions’ policies). The Bill & Melinda Gates Foundation is a leader among philanthropic organizations in formulating and implementing open access policies. Beginning in January 2017, the Gates Foundation’s Open Access Policy requires immediate open access (“Gold” access) for all publications and underlying data generated by authors that it supports .Private funders have taken a variety of approaches to promoting open access, such as increasingly requiring either full open access or archived manuscripts as a condition of continued funding . This issue warrants further attention as funders and journals continue to negotiate around access permissions. The Wellcome Trust has a similar policy, encouraging immediate open access but allowing a six-month delay. Both the Wellcome Trust and Gates Foundation have provided dedicated funding to support open access fees imposed by journals where appropriate, and prefer the unrestricted Creative Commons-BY license (https://creativecommons.org/licenses/by/4.0/). More recently, both the Gates Foundation and the Wellcome Trust took the additional step of partnering with F1000 to establish publishing platforms for their grantees.Many journals already have open access options that comply with the Gates Foundation policy, but some high-profile journals such as14. Additionally, 20% of researchers were unaware whether their funders had an open data policy and most researchers welcomed additional guidance on their funders’ openness policies14, suggesting the need for increased education and support. One facet of the Gates Foundation solution to this problem is a new service called Chronos. The Chronos service guides users through submission to services that are compliant with Gates’ policy, automatically pays open access fees, and archives manuscripts on PubMed (https://youtu.be/lweC1BajBBY). The Gates Foundation expects to scale Chronos to additional funding organizations (https://chronos.gatesfoundation.org/dynamic.aspx?data=article&key=13-What-is-Chronos&template=ajaxFancyArticle).While this represents real progress, these policies can be a source of confusion for researchers. In a recent survey of over 1,000 researchers by figshare and Digital Science, 64% of researchers who have made their data open could not recall what licensing rights they had granted on the data arXiv orbioRxiv, before publication. Some journals, such asPeerJ, also have their own pre-print option46. PubMed Central and European PubMed Central also provide open full text archives. The precedent set by these large funders has established an infrastructure and leadership base that smaller funders may be able to leverage in the development and advancement of their own open access policies. Supported by the Laura and John Arnold Foundation, the Center for Open Science also has developed implementation guidelines for funders interested in establishing transparency and openness policies39. Like the TOP journal guidelines, the TOP funder policies are tiered to allow funders to implement more stringent standards over time. Starting in March 2017, the U.S. NIH has begun encouraging investigators to cite preprints or draft (non-peer-reviewed) manuscripts as part of their funding applications47.The leadership of funders has led several journals to allow authors to self-archive manuscripts on preprint servers, such asBoth governmental and private funders have undertaken significant policy changes to mandate open access to data sets and publications. Funders are generally moving towards more open access, mandating or encouraging researchers to publish in open access journals, paying open access fees, and requiring manuscript archival when researchers publish in more restrictive journals.45, and requires all grantees to deposit articles and/or manuscripts in this open repository within twelve months of publication (https://publicaccess.nih.gov/policy.htm). The Gates Foundation and Howard Hughes Medical Institute have leveraged the NIH’s investment by requiring their own grantees to archive manuscripts in PubMed . Gates has gone one step further on open access, requiring all publications to be immediately available in open access “Gold” format . The Gates Foundation has also developed tools to assist its grantees with compliance with these new open access policies (https://youtu.be/lweC1BajBBY).Large funders are leading the drive towards open access. NIH spends roughly $4.5 million on PubMed Centralhttps://creativecommons.org/licenses/by/4.0/).As major funders increasingly mandate open access, more journals are providing open access options for authors. Many journals provide Creative Commons copyright options, providing a uniform set of standards. The increased adoption of Creative Commons licenses by journals, especially unrestricted CC-BY licenses, reduces the barrier to adoption of open and transparent sharing permissions . At least 31 journals recommend or require authors to submit their plasmids to the Addgene repository (https://www.addgene.org/deposit/pre-publication/). Addgene performs sequencing to verify submission quality , and requires each contributor to provide the same types of information in a uniform format, making the database easily searchable and comparable.48. Similar to the development of reagent standards described above, updated data standards will require multi-stakeholder collaboration within the community of practice, harnessing existing standards where possible and harmonizing divergent practices where appropriate.The Addgene approach works well for plasmids, which consist of a relatively limited number and size compared to high- throughput, whole genome sequencing data sets. As next generation techniques become more widespread, data standards will become even more important. These data standards include metadata , data fields, and file formats. With data standards, large data sets become much easier to download and interpret, because users do not have to spend valuable and expensive computational time modifying existing analysis tools to fit each new data set. Researchers have proposed a series of metadata checklists for high-throughput studiesPost-publication review. Scientific review is an ongoing process that continues well after peer-review and publication. The broader scientific community may identify issues that were not highlighted by the peer reviewers, and other researchers may attempt to reproduce a study on their own. As the post-publication review process may require experimentation, it warrants dedicated resources.eLife, an open access journal supported by the Howard Hughes Medical Institute, Max Planck Gesellschaft, and the Wellcome Trust49. Two of these five studies successfully reproduced the original findings, one study did not, and two attempts were inconclusive. Since the project seeks to reproduce approximately 50 papers, conclusions about the Project’s reproducibility rates at this early stage  would be premature. An earlier project, Reproducibility Project: Psychology, attempted to reproduce 100 original psychology findings, successfully reproducing one-third to one-half of the results50. Another open access publication,F1000Research, established thePreclinical Reproducibility and Robustness Channel as a platform dedicated to reproducibility of published papers (https://f1000research.com/channels/PRR).Despite the time commitment and added value to science, the research community typically does not reward post- publication review. Historically, funding agencies and tenure boards do not tend to reward results reproducibility studies, and researchers can have trouble convincing journals to review and accept such manuscripts. However, stakeholders from different sectors now are dedicating resources to results reproduction. The Laura and John Arnold Foundation currently is funding a cancer biology results reproducibility study as part of its Reproducibility Project series. The first five attempts to reproduce papers as part of this effort were published in January 2017 in the journal51. Similarly, journals do not always have a clearly-defined retraction process that mirrors the submission and peer review processes. Much like the stakeholder discussions on study design, cell line authentication, and open access, the retraction process is an important topic that warrants engagement by the research community. The Committee on Publication Ethics has established best practices for Retraction Guidelines52, which may provide an opportunity for this discussion.Researchers attempting to raise concerns to editors about irreproducible or incorrectly analyzed results found in published articles describe many barriers to the process of raising these concerns, including lack of clarity and transparency from journals in the post-publication review processPubMed Commons andPubPeer, provide an informal mechanism to facilitate post-publication review and results reproduction attempts by providing a discussion forum for researchers to openly discuss scientific publications. Discussions on these platforms can occur much faster than the pace of published technical commentaries in journals, and provide opportunities for more scientists to contribute. Last year, researchers undertook a widespread deployment of the automated statcheck algorithm on nearly 700,000 experiments from over 50,000 papers, and automatically generated comments on PubPeer for each paper53. This automated tool helps researchers identify papers that deserve further review and discussion about solutions, such as retraction or publication of counter studies. Discussions on open blogs are a double-edged sword. Whereas rapid turnaround and informal discussion can stimulate productive scientific debate, unmoderated discussion can also lead to unwarranted criticism of legitimate studies. In contrast, technical commentary in journals is refereed by an editor who can help organize and moderate the discussion.Websites, likewww.retractionwatch.com), bring publicity to reproducibility and retraction news, which can galvanize the scientific community to action. For example, replicability of the initial paper describing the NgAgo genome editing technique has been the subject of fierce debate in the community wherein researchers described their difficulties in reproducing the paper’s claims on internet and scientific news sites. The technique drew so much attention that over 100 researchers attempted to reproduce the technique in the first few months after publication, but less than 10% were successful54. The controversy resulted in three peer-reviewed publications, all of which documented a failure to reproduce the original study, and researchers now are trying to understand the reasons for irreproducibility55.The sheer volume of published research increases the difficulty of identifying and tracking publication errors. Science journalism is another tool that can improve reproducibility. Science reporters, such as the authors of Retraction Watch (56. Researchers armed with this database can avoid using retracted work as a (shaky) foundation for new studies, thereby increasing their chance of success. By reading about reproducibility and retraction news, researchers can learn about the common pitfalls that can cause retractions and new resources available to help them improve the reproducibility of their work, such as the initiatives described in this report. However, highly-visible retractions are a potential threat to public confidence and support for science, as the lay public reads more about retractions and irreproducibility. This further highlights the urgent need for the scientific community to act on the initiatives described in this report and make meaningful improvements to reproducibility.Retraction Watch also partners with the Center for Open Science to generate a database of retractions, as some retracted articles still are cited frequently after retractionIrreproducibility is a serious and costly problem in the life sciences. Measured reproducibility rates are shockingly low, requiring significant effort to solve this problem. Many stakeholders now recognize the importance of reproducibility and are taking steps to develop and implement meaningful policies, practices, and resources to address the underlying issues. The lessons learned from these early efforts will assist all stakeholders seeking to scale up or replicate successful initiatives. The research community is making progress to improve research quality. By prioritizing the strategies outlined in the Report, stakeholders in life science research will continue to make progress in improving reproducibility and in turn have a profound positive impact on the subsequent development of treatments and cures.However, the authors would be remiss if we ignored a transcending challenge facing the research community and their willingness to voluntarily accept these positive steps in addressing reproducibility: the current rewards system in academia, including constant pressure to obtain grants and publish in “high impact” journals. The research culture, particularly at academic institutions, must seek greater balance between the pressures of career advancement and advancing rigorous research through standards and best practices. We believe that the many initiatives described in this Report add needed momentum to this emerging culture shift in science, but additional leadership and community-wide support will be needed to better align incentives with reproducible science and effect this change.Continued transparent, international, multi-stakeholder engagement is the way forward to better, more impactful science. GBSI calls on all stakeholders – individuals and organizations alike – to take action to improve reproducibility in the preclinical life sciences by joining an existing effort, replicating successful policies and practices, providing resources to results reproduction efforts, and/or taking on new opportunities.In its leadership role, GBSI will:work with journals and funders to encourage policies that increase rigor, accountability and open access to data and methodologies;lead the effort toward improving the validation of reagents—particularly cells and antibodies— and work with the research community to explore other scientific areas (e.g. stem cells and synthetic biology) where a greater emphasis on development of standards and best practices are needed to ensure quality and advance discovery;ensure high quality, accessible online training modules available to both emerging and experienced researchers who are eager to improve their proficiencies in new and evolving best practices; andcontinue to track reproducibility efforts through the Reproducibility2020 Initiative.The preclinical research community is full of talented, motivated people who care deeply about producing high-quality science. We are optimistic about the potential to improve reproducibility, and look forward to contributing to the effort. This is a carefully considered, well written, and comprehensive overview of the numerous causes of irreproducibility and the many ongoing efforts to address them. This manuscript also provides a set of useful actionable recommendations for researchers, funders, journals, and other stakeholders to improve the rigor and reproducibility of research. Below are specific comments that I hope the authors will find useful for revising and improving their paper.funders of GBSI and this report mentions ATCC a couple of times. The mentions are appropriate, but the GBSI/ATCC relationship should be clearly disclosed in the COI.ATCC is one of the main[Abstract and Introduction]https://www.ncbi.nlm.nih.gov/pubmed/16510544,http://www.the-scientist.com/?articles.view/articleNo/16604/title/Microarray-Data-Stands-Up-to-Scrutiny/,http://www.nature.com/ng/journal/v41/n2/full/ng.295.html,http://iai.asm.org/content/78/12/4972.full,http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0040028,http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020272,http://elpub.scix.net/data/works/att/001_elpub2008.content.pdf). Moreover, many efforts to improve reproducibility are significantly older than 2012 . Would be good to explicitly acknowledge this. Both the abstract and introduction mention the 2012 Amgen report as the beginning of attention to reproducibility. Without a doubt, the Amgen and Bayer headlines have led to a spike of attention and discussion; however the reproducibility issue is not a new problem. Inability to repeat the work of others is as old as science itself and much has been previously written regarding this issue . It extrapolates to all of US Biomedical Funding from a few estimates of irreproducibility in specific fields. I know of no quantitative research that evaluates reproducibility of published basic research in zebra fish or drosophila communities. If reproducibility problems are greater in cancer, human cell lines, and other research fields, the overall scale of the reproducibility problem across all biomedical research could be smaller. As has been publicly discussed after the PLOS Biology publication [4], the estimate of $28B cost of irreproducible research is on shaky ground [Laboratory protocols]Protocol Exchange fromNature/Springer which is a protocol repository that was started over a decade ago to improve the reporting of methods. This section should probably mention theeLife recently included them in their author guidelines to encourage scientists when appropriate to submit new method details to Bio-protocol in parallel with their eLife manuscript submission. The authors might also want to include a mention of Bio-protocol, a journal devoted to increasing reproducibility. Though a selective peer-reviewed journal rather than a repository, Bio-protocol is also connecting to journals and[Reporting and review]Dryad andfigshare. Journal policies regarding data sharing are critical and this overview of the genomics community journal policies from Heather Piwowar and Wendy Chapman is relevant:http://elpub.scix.net/data/works/att/001_elpub2008.content.pdf. In the data reporting section, I recommend adding a brief discussion of data repositories such atdata policy from the Public Library of Science is an important step in improving reproducibility of published work. Also, the explicit Related to the data policies, sharing code and software from computational pipelines used to analyze the data is critical. Perhaps add a mention of policies encouraging proper reporting and sharing of code/software?[Reporting and review] There are important experiments happening with open review from publishers such as F1000 Research, EMBO, BMJ, PeerJ and others. Transparent publication with review/author response history can be helpful for reproducibility as readers can see reviewers’ concerns and that can help to discern which parts of the paper are more or less trustworthy.Transparency In Authors' Contributions And Responsibilities To Promote Integrity In Scientific Publication.) Another relevant proposal is for the adoption of CRediT (Contributor Roles Taxonomy) by publishers. , but I don’t think I saw it in this paper.[Reporting and review: preprints] As above for open access, I am a huge fan of preprints but am unsure how they fit into the push for greater reproducibility.  Preprints, of course, shorten publication delays, facilitating communication and speeding up research. However, preprints are not peer-reviewed, do not go through conflict-of-interest checks, data/method reporting compliance checks, and so forth. At scale adoption of preprints in biology is welcome for many reasons, but not exactly due to more rigor and higher reproducibility.  (Minor note: the use of “preprint” versus “pre-print” is inconsistent in this paper. Please remove the extra dash.)For funders, there is a recommendation to “Enact policies requiring study design pre-registration”. I am on the steering committee for COS’s pre-registration initiative and support this effort, but I am not sure that “requiring” pre-registration widely is appropriate. This will depend on the funder and specific research grant. For example, in the case of method development and highly explorative grants, pre-registration is unlikely to be productive. How about “encourage where appropriate” instead of “require”?Require authors to link to version-controlled protocols”. Again, “require” is a strong term. In certain cases, it may be better to share a protocol directly as part of the publication . A more general “encourage or require detailed reporting of protocols” may be more appropriate. For journals, there is a recommendation to ““Irreproducibility is a serious and costly problem in the life sciences. Measured reproducibility rates are shockingly low, requiring significant effort to solve this problem.” [Conclusion] I very much agree with the first sentence in that irreproducibility is a serious problem. However, is the reproducibility rate “shockingly” low? What is that rate for biology in general? As discussed above, 50% may be the number for some fields but not for others. More importantly, what rate are we aiming for? 70%? 90%? If all of the action items recommended in this report were followed, what rate would we end up with? Is our current level of reproducibility better or worse than it was 30 years ago? What is the optimal reproducibility rate from society’s perspective? I don’t have the answers to the above questions. We need a lot more data to make informed statements about the levels of reproducibility over time. It is terrific that we are discussing this issue and the initiatives to address the problem, but I urge caution in editorializing about whether today’s reproducibility levels are a “crisis” or are “shocking”. Science is hard and because it is pushing the boundaries of knowledge, we will never be at 100% of published research being reproducible. We can and should do a lot better, hence all of the initiatives, but it will never be 100%.[General thoughts] As I mention in #11 above, with the exception of a few efforts from Science Exchange and the Center for Open Science, we have very little data on the reproducibility issue. The authors may want to include in their discussion the need for more quantitative studies about replication and reproducibility over time. We need ways to assess the various initiatives and to measure whether they are in fact improving the overall reproducibility levels of published research. Also, most of the recommendations and discussion in this Report are focused on the design, execution, and publication steps of the research cycle. However, given the complexity of research and the fact that we will never attain 100% reproducibility, efforts aimed at post-publication opportunities to improve reproducibility may be particularly effective. Perhaps we should pay more attention not just to preventing mistakes, but to ways to correct and improve papers, long after publication.recent proposal argues for rethinking of “retractions/corrections” in favor of "amendments" to increase post-publication evolution and improvement of work. This Report mentions post-publication review and retractions, but there are other promising efforts in this phase. Versioning, as implemented on F1000Research and bioRxiv, has great potential. There is a need for technologies that automatically connect readers to corrections and discussion on the papers that they have in their libraries. Crossmark from Crossref is a great initiative aimed at making corrections discoverable. Also, an interesting -------------------------------------   I would like to stress that I thoroughly enjoyed this report and am grateful to the authors and GBSI for their efforts to improve the research enterprise for the benefit of scientists and the public. The authors should feel free to ignore any of the above suggestions if they disagree.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Freedman and colleagues present a narrative review on current efforts underway to improve reproducibility in preclinical biomedical research.  They begin by summarizing the extent of the problem and noting that quality checkpoints are either used in disparate points of the research cycle or used only sparingly.  They identify key sources of irreproducibility as poor study design and analysis, inadequately authenticated reagents and reference materials, inadequately documented laboratory protocols, and inadequate reporting and review.   They describe the important roles of many stakeholders, including funders, researchers, research institutions, industry, foundations, professional societies, and the public.  The authors proceed to describe many efforts already under way including new funder requirements, journal guidelines, enhanced training opportunities, programs to enhance standards development and authentication checks, protocol repositories, improved reporting platforms, open access policies , data standards, and post-publication review.  They conclude with a “path forward” that they call the “Reproducibility2020 Action Plan” that includes specific recommendations for funders, researchers, institutions, journals, industry, foundations, and the public. Thoughts and comments:The paper is interesting, well-written, and well-documented.  I appreciated the many web links that take the reader directly to interesting sites.https://doi.org/10.1371/journal.pmed.0020124).  Ioannidis there suggests that exploratory research was highly vulnerable because of small sample sizes, overly flexible designs, and biased designs (e.g. with lack of randomization and proper masking).The authors suggest that the current crisis begins with the Amgen findings (Reference 2).  While that was a defining moment, I wonder whether it’s also worth mentioning that contemporary discussion about false research findings dates back at least as far back as Ioannidis 2005 , that may be good for the enterprise – it would be better to do fewer properly powered experiments than to do too many woefully underpowered experiments.Table 1: I commend the authors for noting that “the chance of an irreproducible finding is much higher than the commonly noted 5% threshold.”  This is widely under-appreciated, even by well-trained scientists.  The authors might consider spelling out that prospective, properly done sample size calculations are critical to overcoming this problem.  The “elephant in the room” is that sample sizes will have to increase substantially, meaning that with constrained funds researchers will be forced to conduct fewer experiments.  But as some have noted .  This is already common practice in some fields (e.g. economics).Table 2: Another consequence for the public is lack of faith in science.  They hear scientists promising the moon, and then nothing happens. Table 2: There is an ethical problem subjecting animals and people to inadequately designed or documented experiments that were doomed to be irreproducible from the beginning.https://www.nap.edu/catalog/21896/fostering-integrity-in-research). Table 2 or elsewhere: NAS just released a report on research integrity in which notes a continuum between frank misconduct  and “practices detrimental to research.”  The authors might want to consider the comments of the report .  The authors note that “stroke research has uniquely improved.”There have been some recent successes in improved rigor, such as in preclinical stroke research.  (For example, seePage 6 – the link didn’t take me directly to “Statcheck software,” though I did eventually find it.Protocols – many leading clinical journals require authors to submit full clinical trial protocols along with the manuscripts.Should it be the responsibility of funders to provide statistical consultation to applicants?Should it be the responsibility of funders to pay for open access and transparency tools? Should funders include dedicated reviews on methodological issues for those applications deemed meritorious by content? Table 3  I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.


International Journal of Molecular Sciences will be retracted. An investigation of the peer review process has concluded that it was compromised via the use of fake reviewer email addresses provided by the authors. MDPI takes the integrity of its review process very seriously. Since bringing the management of the peer review process in-house in 2006 ,An internal investigation has concluded that for these two papers the procedures were not correctly followed, for which we apologize to the readers of the journal. The investigation covered all MDPI journals and these were the only problematic cases discovered.We believe that the vast majority of authors submit manuscripts in good will, however we remain vigilant for suspicious activity and strive to have robust procedures in place to ensure that such cases can be discovered prior to publication.


PLOS Medicine Editors discuss the role of preprints in human health research and propose a 3-part framework for ensuring benefit.In this month’s editorial, the  PLOS Medicine. In an analysis of published articles and those posted on preprint servers, the authors found that preprint posting increased during the 2015–17 outbreak of Zika in the Americas relative to the 2014–15 Ebola outbreak in West Africa, and that preprints, while still used for only a small proportion of papers, provided much earlier access to scientific findings [Sharing of preprints—scientific manuscripts that are posted in a publicly accessible, online repository before peer review for journal publication—can accelerate access to information in infectious disease outbreaks, according to an article published earlier this month by Michael Johansson and colleagues in For medical journal editors who agreed in late 2015 and early 2016 that authors should not be penalized for sharing results in public health emergencies , these rPLOS Medicine has long permitted the posting of preprints for research articles submitted to the journal, and this remains our policy. For the most part, posting of preprints has been occasional, undertaken on the initiative of authors without the journal’s involvement. In the coming months, PLOS journals that handle research in more basic biological fields will begin facilitating preprint posts of journal submissions, with author approval, on the bioRxiv preprint server operated by Cold Spring Harbor Laboratory [PLOS Medicine will not initially be offering transfer of submitted manuscripts to bioRxiv as preprints. Because the journal’s scope encompasses clinical and policy-related research, we believe it is important now to establish appropriate standards for the early sharing of research that has potential implications for human health but has not been peer reviewed. As preprint servers for clinically focused research become more widely available and preprint posting becomes more routine and more closely integrated with journal publishing, editors of medical journals should join the conversation about best practices for preprint sharing in medical research.boratory . PLOS MeIn calling for wider use of preprints, Johansson and colleagues note the importance of protecting participant confidentiality, ensuring that posted research was ethically conducted, and guarding against misinterpretation of results that have yet to undergo peer review, either by traditional journal routes or through newer, more public venues . In mediA constructive solution will reflect the legitimate priorities—and depend on the engagement—of several groups. We propose a framework of three broad categories: transparency in reporting; clarity about the role of a preprint versus a peer-reviewed journal article; and responsibility for safety.First, preprints of relevance to human health should incorporate the best practices that medical journals have developed for transparency in reporting. Funding sources, author competing interests, and involvement of funders in creating or deciding to post the preprint should be declared. For clinical trials, preprints should include reference to the trial’s registration in a WHO-approved database  and shouSecond, participants in all stages of the information sharing process should maintain the distinction between a preprint and a peer-reviewed article. For example:Preprints should be clearly marked , in all accessible forms  as not peer reviewed nor accepted for journal publication. Reporters and media outlets that cover health research should be informed by research and publishing communities, and in turn should undertake to educate the public, that preprints are intended for researchers familiar with limitations of the study methodology, and that results or conclusions may change during peer review, which often includes assessment of statistical methodology.Authors and institutions should not seek media attention for preprints, and if contacted by the press should acknowledge that the content has not been peer reviewed. If the preprint does receive press coverage, authors should later apprise reporters of any changes between preprint and published article that substantially alter clinical or policy implications, and should understand that journals may also alert the media to such changes.Preprint servers should indicate when journal publication has occurred and provide a link from the preprint to the published article. Journal articles preceded by a preprint should include a reference to the preprint. Journal editors may ask authors to summarize substantive differences between the preprint and peer-reviewed version of their work.These practices will clarify relationships between preprint and published versions and discourage unwarranted or uncritical application of information contained in preprints.Third, preprints should not be posted when the reported information could be misapplied to pose a significant threat with potential consequences to health and safety. No single list or algorithm can identify all situations in which potential errors or misinterpretations are likely to cause harm that could be avoided by not posting. Sound judgment by authors, journal editors (when involved in screening or facilitating preprint posts), and preprint services must inform the decision to post a preprint. For some manuscripts, timely expert review before public sharing may remain the optimal approach. In all cases, preprints must adhere to established standards for the ethical conduct of human research, including the protection of participant confidentiality.These points do not constitute journal policy, and cannot; their implementation goes beyond the remit of journal editors to involve the broader community of journalists, preprint service providers, institutions and researchers themselves. Nor are these points exhaustive; as in other areas of research it will also be important to address such practical issues as copyright licensing for preprints and community standards around “scooping” when a journal submission is antedated by a preprint on a similar topic by different authors.As opportunities for preprint sharing advance, medical journals should endeavor to ensure that the resulting changes benefit human health and support the public understanding of medical research. We hope these ideas will promote wider discussion among researchers and research participants, preprint posting services, journalists and journal editors. We welcome your comments.


Open data is a vital pillar of open science and a key enabler for reproducibility, data reuse, and novel discoveries. Enforcement of open-data policies, however, largely relies on manual efforts, which invariably lag behind the increasingly automated generation of biological data. To address this problem, we developed a general approach to automatically identify datasets overdue for public release by applying text mining to identify dataset references in published articles and parse query results from repositories to determine if the datasets remain private. We demonstrate the effectiveness of this approach on 2 popular National Center for Biotechnology Information (NCBI) repositories: Gene Expression Omnibus (GEO) and Sequence Read Archive (SRA). Our Wide-Open system identified a large number of overdue datasets, which spurred administrators to respond directly by releasing 400 datasets in one week. Advances in sequencing and other biotechnologies have led to an explosion of biological data. PLOS journals require authors to "make all data underlying the findings described in their manuscript fully available without restriction" + for GEO, and SRX[0–9]+ for SRA. For each candidate accession number, Wide-Open first checks GEOmetadb  for GEO,Specifically, for GEO, Wide-Open looks for strings such as "Could not find a public or private accession," which signifies an invalid accession number, as well as strings such as "is currently private," which signifies that the dataset is private. For SRA, the process is similar. The details can be found in our open-sourced code.Wide-Open identified a large number of overdue datasets in GEO and SRA. As of March 2017, Wide-Open has identified 84 overdue SRA datasetsas. Next, we plan to contact SRA and work with them on verification and release of these datasets as well.The time lag between submission and release has also steadily risen . GEO dathttp://text.soe.ucsc.edu/progress.html). Through our collaborators, we begin to have access to many more full-text articles on which we plan to run Wide-Open next. The number of private datasets is rather large. For example, GEO currently has over 10,000 datasets that remain private. We expect that many more overdue datasets could be identified with access to additional full-text articles.While the initial progress is promising, much remains to be done. We need full-text access to identify published datasets, which limits our current monitoring to the open access subset of PubMed Central (PMC). As of February 2017, this subset contains about 1.5 million papers, which is a small subset of PMC (4.2 million) and a fraction of PubMed (26 million). There are various ways to substantially increase the number of full-text articles for monitoring thanks to the open-access movement championed by this journal and others. Publishers are increasingly open to granting text-mining licenses . We will host a service to keep monitoring publications and identifying overdue datasets (https://wideopen.github.io/datawatch/). We also plan to extend Wide-Open to cover more repositories and implement more fine-grained audits . Extending Wide-Open to a new repository consists of 3 simple tasks: creating regular expressions for dataset identifiers, identifying the web API for dataset access, and adapting the query-result parser to distinguish between invalid UIDs, datasets that have been released, and datasets that remain private. We envision Wide-Open as a collective project to engage the research community and help advance the open-data movement.Wide-Open is available under an open source license at GitHub (S1 Datahttps://www.ncbi.nlm.nih.gov/geo/summary/summary.cgi?type=history.Obtained from (CSV)Click here for additional data file.S2 DataComputed using Wide-Open extracted references, GEOmetadb and queries against GEO web interface as described in the paper.(CSV)Click here for additional data file.S3 DataComputed using data from GEOmetadb.(CSV)Click here for additional data file.S4 Data(XZ)Click here for additional data file.


Journal of Applied Clinical Medical Physics, we conclude the publication of our eleventh volume. In the past several years we have seen a profound growth in the JACMP. In 2008, the last year that my predecessor, Michael Mills, was Editor‐in‐Chief, we received a total of 142 submissions, or an average of 35.5 manuscripts per quarter. This number was exceeded by almost 25% during the first three quarters of this year. We are well on our way to receiving over 200 submissions for this calendar year. Our rejection rate remains at approximately 45%, so we anticipate significant growth in the number of manuscripts published in the JACMP.With the publication of this issue of the We are not anticipating a large number of changes going into next year. The following is one important change we are instituting for manuscripts submitted effective January 1, 2011: if any humans are involved in a study, appropriate approval from the author's institutional review board must be obtained. Authors need to indicate in the manuscript that such approval has been obtained. This approval also includes use of patient data such as treatment plans. This requirement is consistent with editorial policies in most other journals. For details about the policy, prospective authors are advised to review the Submission Guidelines.In addition, we are trying to limit the size of the JACMP. Rather than increasing our rejection rate, we are encouraging a limit 5000 words for each article. This length restriction will be a recommendation, not a requirement. Associate Editors and reviewers will be reminded of this length restriction when a manuscript is sent out for review. Authors of manuscripts that are significantly longer than 5000 words will be encouraged to restrict the length, especially by reducing the length of the Discussion section.Finally, when mentioning Associate Editors and reviewers, I would like to acknowledge their difficult and time‐consuming work in maintaining the quality of manuscripts we publish in the JACMP. Names of Associate Editors can be found below, and names of reviewers can be found in a Supplementary File.Associate Editors (including Guest Associate Editors) for Volume 11 include the following: Nzhde Agazaryan, Salahuddin Ahmad, John Antolak, Sam Beddar, Pat Cadman, Marco Carlone, Nathan Childress, Geoff Clarke, Laurence Court, Luca Cozzi, Larry DeWerd, Bill Erwin, David Followill, Kent Gifford, Michael Gossman, Rebecca Howell, Geoff Ibbott, Jennifer Johnson, Stephen Kry, James Marbach, Mary Martel, Osama Mawlawi, Charles Mayo, Moyed Miften, Eduardo Moros, Firas Mourtada, Ben Nelms, Jennifer O'Daniel, Tinsu Pan, Niko Papanikolaou, Matt Podgorsak, Joann Prisciandaro, Jim Rodgers, John Rong, Yi Rong, Isaac Rosen, Surendra Rustgi, Bill Salter, Mehrdad Sarfaraz, Jeff Shepard, Chengyu Shi, Alf Siochi, Eric Slessinger, Jason Stafford, Frank Van Den Heuvel, Lu Wang, Chuck Willis, Al Zacarias, and Ron Zhu.Thanks are extended to these individuals and reviewers, and especially to the authors, without whose contributions we would have no journal.George Starkschall, PhDEditor‐in‐ChiefNovember 15, 2010


Nutrition Journal, one of BioMed Central’s open access journals. Diet is among the leading risk factors for morbidity and mortality globally [This editorial introduces a revised and refreshed scope for globally . This siNutrition Journal will prioritize manuscripts reporting on novel and high-impact research shedding light on factors that influence eating patterns among diverse populations, and studies that elucidate relationships between eating patterns and health and disease outcomes. We also welcome intervention research, including controlled trials but also natural experiments that uncover successes and challenges in implementing strategies to improve eating patterns among populations, including interactions between policies and programs and contextual factors. Within these areas, we invite manuscripts reporting on research conducted with various populations across the lifecycle and in diverse settings. Considered is research within public and population health, as well as in clinical settings and among individuals with chronic disease risk factors or endpoints.With its refreshed scope, Dietary assessment is fundamental to much nutrition research. With the refreshed scope, the journal places a more explicit focus on methodological considerations. We encourage authors to peruse publicly-available resources –4 relateIn all papers reporting on studies including the measurement of dietary intake, we encourage perusal of the checklist associated with the nutritional epidemiology extension of the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement, known as STROBE-Nut . AlthougStudy protocol manuscripts will be considered in light of the aim of increasing transparency of methodology such that findings can be interpreted appropriately. Publication of protocols is also intended to contribute to the harmonization of methods across studies so the resulting evidence can be synthesized in a meaningful way. The journal thus welcomes the submission of manuscripts describing protocols for cohort studies and controlled trials, including studies designed to provide baseline data to enable leveraging natural experiments  to offer novel insights. As above, within such papers, we seek detailed descriptions of the dietary methods and measures used to assess dietary exposures and outcomes and the considerations weighed in arriving at these methods and measures, as well as the planned use of statistical approaches to mitigate error in dietary data. Protocols for randomized controlled trials should be accompanied by the checklist corresponding to the SPIRIT Statement , 11.The journal will continue to consider systematic reviews and meta-analyses that adopt robust methods for the identification of relevant literature and abstraction of results. Authors of such papers should consult the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)  statemenFinally, given that publication in the peer-reviewed literature is only one of many approaches to dissemination with reach limited to particular audiences, the journal also encourages the submission of papers advancing the science of knowledge translation in the area of nutrition evidence, both to health professionals and members of the public.With our refreshed scope, we are embracing a theme of holism. The journal has a strong tradition of publishing research reporting on the impact of specific dietary constituents on risk factors for disease. We will continue to consider such studies provided a clear rationale is articulated, but encourage researchers to consider eating patterns more holistically given increasing recognition that it is the combination of foods and beverages we consume rather than particular nutrients or other dietary components in isolation that influence our health . Thus, wGiven our focus on advancing our understanding of nutrition and its influences in populations, the journal does not publish animal studies.Nutrition Journal is an open access, broad scope journal that focuses on human and population nutrition. All articles published by Nutrition Journal are made freely and permanently accessible online immediately upon publication, without subscription charges or registration barriers. Nutrition Journal operates an open peer-review system, which increases transparency and accountability through the identification of reviewers. Additional details regarding article types and submission guidelines can be found at https://nutritionj.biomedcentral.com/.Nutrition Journal. Our vision is to create a journal recognized for publishing high-quality human nutrition research with a focus on transparency and robustness of methods, particularly those used for measuring dietary exposures and outcomes and related dietary phenomena.We are very pleased to invite you to submit your research manuscripts to


Journal of Comorbidity was launched in 2011 and has since become established as a high-quality journal that publishes open-access, peer-reviewed articles, with a focus on advancing the clinical management of patients with comorbidity/multimorbidity. To further enhance research quality and reporting of studies in this field, the journal is now offering authors the opportunity to publish a summary of their study protocols – a move designed to generate interest and raise awareness in ongoing clinical research and to enable researchers to detail their methodologies in order that replication by scientific peers is possible.The Study protocols are a fundamental component of any study and provide insight into the thinking behind the study and how it will be conducted prior to its initiation. Protocols should meet the information needs of all stakeholders, including researchers, funding agencies, ethical reviewers, regulators, medical journals, and systematic reviewers . They shPublishing study protocols serves several important purposes. Publication increases research quality and transparency, encourages communication and collaboration between research teams, helps to disseminate contemporary ideas about study design and data analysis, avoids research duplication, and may assist study recruitment.post-hoc revisions of the study goals and methodologies – a problem that appears to affect the reporting of both observational studies and randomized controlled trials.The availability of a published final protocol allows subsequent authors and reviewers to cross-reference the study details, assists in presenting the final results, aids systematic evaluation of the data, and discourages reporting bias. Importantly, it enables readers to compare the original aims and design of the study with the final reported outcomes, thereby reducing the potential for unexplained Publication of study protocols in peer-reviewed journals rightfully engenders the expectation that the results will eventually be disseminated, and encourages investigators to report their findings in a timely and appropriate way. Last, but not least, we believe that publication of study protocols recognizes the substantial amount of work that is required for their development, affording research professionals, who may otherwise remain unseen, an opportunity to promote their work.Journal of Comorbidity will peer review protocols unless the study has received approval from an institutional review board (IRB)/ethics committee and has received peer-reviewed grant support from a major extramural funding bodyThe Journal of Comorbidity endorses the SPIRIT  Statement [The tatement . AuthorsProtocols that describe the design of planned or ongoing research will be considered for publicationProtocols for any study design will be considered, including observational, qualitative, exploratory studies, experimental studies, and systematic reviews. We encourage the submission of protocols for studies that will evaluate specific aspects of clinical trial designStudy protocols should generally provide a detailed account of the hypothesis, rationale, contextual background, design/methodology, statistical considerations, and analysis of the proposed researchStudy protocols should be presented with sufficient detail to enable replication of the methods by otherswww.jcomorbidity.com).Full details of the requirements for submitting a protocol are described in the journal’s guidelines to authors, which can be found online .To submit your protocol to the Journal of Comorbidity is an international, open-access, peer-reviewed journal for the pathophysiology, diagnosis, prevention and management of patients with comorbidity/multimorbidity. The journal currently publishes original clinical and experimental research articles, treatment guidelines/policies, editorials, commentaries, and critical review articles, as well as proceedings of congresses. Preference is given to articles that advance the clinical management of patients. The Editorial Board also welcomes ideas and suggestions for special issues dedicated to a unique theme. The journal provides immediate, permanent open access to its content on the principle that making research freely available to the public supports a greater global exchange of knowledge.The


Molecular Systems Biology is partnering with the new open‐access journal Life Science Alliance, an initiative by three major not‐for‐profit science organizations. An informed transfer process was established with our partner to provide authors with clear editorial commitments and reduce the overall burden on reviewers. Portability and modularity are good engineering principles that are applied to build systems of increasing complexity with compatible components that do not need to be reinvented each time. These principles are, however, not always applied in scientific publishing where competing publishers tend to operate as independent ‘silos’ and have disappointingly little incentives to develop common standards for portable information exchange. This is particularly detrimental to the efficiency of the peer‐review process, where cohorts of reviewers are asked to spend a significant amount of time assessing the same manuscript over and over, as it goes through repeated submissions at a succession of journals. For the global research community, this results in a considerable waste of resources and slows down the publication process in the most frustrating way.http://nprc.incf.org/) to enable transfer of reviews among participating journals. In addition, many journals offer authors the possibility to transfer their manuscripts and reviews to journals published by the same publisher. While these initiatives are welcome, they may not have always been as transformative as expected, in particular when applied across publishers. For one, it is difficult to change the behaviors of scientists and editors who often prefer to start the process from scratch rather than embarking on the reinterpretation of reviews written for another journal. But it may also reveal that portable review is only part of the solution. From our own experience with manuscript transfers within the EMBO Press journals, including Molecular Systems Biology, two factors are decisive to authors when they are considering whether to forward their manuscripts to another journal: obtaining a clear editorial commitment from the potential recipient journal; and keeping control over the fate of their work.The idea of making peer‐review portable across journals is not new. Ten years ago, the neuroscience community launched the Neuroscience Peer Review Consortium , jointly published by EMBO Press, Rockefeller University Press and Cold Spring Harbor Laboratory Press. Together with the journals participating in this alliance, Molecular Systems Biology has established an informed transfer process that goes beyond the standard transfer of manuscripts with portable reviews. For manuscripts that cannot be accepted in Molecular Systems Biology, we will consult with the editors of Life Science Alliance before sending our decision. We will explicitly offer a transfer to authors if we obtain a strong commitment from Life Science Alliance. For studies that were peer‐reviewed at Molecular Systems Biology, the commitment of Life Science Alliance will be based on the available reviews and will include a set of clearly scoped revisions without the need to involve additional referees. Importantly, our authors will remain in control of the process: They can opt out of the informed transfer system at submission stage, and accepting or declining the offer to transfer is entirely up to them to decide.We are therefore delighted to announce the partnership of Life Science Alliance “module” inherits many of the attributes from the partnering publishers. Policies and best practices that are already familiar to Molecular Systems Biology authors and readers are thus implemented at Life Science Alliance, including high standards for the quality of the peer‐review process, integration with preprint submission, protection against “scooping”, no reformatting required at submission, and publishing source data alongside figures. Life Science Alliance of course also accepts direct submissions, which undergo transparent peer review.The aim of this partnership is to accelerate the publishing process, and to make it more efficient, without sacrificing quality. Therefore, following good modular practice, the new Molecular Systems Biology, The EMBO Journal, EMBO Reports, EMBO Molecular Medicine, Journal of Cell Biology, Journal of Experimental Medicine, Journal of General Physiology, Genes and Development, and Genome Research. In times when there is a risk of monopoly by few publishers dominating the whole landscape, we feel proud to be part of an alliance, whose members have put aside the inherent competition among their individual journals to join forces in the interest of the research community.The alliance comprises nine major journals, including


JACMP begins with defining our vision of what a 21st century open‐access journal should look like. Yes, money is very tight. Still, the consensus is that the JACMP is serving the needs of the community and we believe the community will find a way to support it. So, given our boundless optimism that the money issues will eventually take care of themselves, let us flesh out some details of what the JACMP would look like if we could have everything we wanted. We will discuss the publication model, financial issues, continuing education activities, and management concerns. We have teams of Section Editors who have volunteered to look after many of the goals discussed below.The future of the JACMP. Thus far, the quotes we have seen for such apps are pretty costly. Still, we have not given up hope that we will find a solution for our users who prefer these platforms to conduct their research. A good application should provide a Title Cover, which could depict any good graph or figure from one of our last year's best paper award winners. Each paper in that issue could be like a bookmark in a pdf format or an electronic book, providing easy access to the user. Between the papers, we can add advertisements. The back page will also have advertisements. This could open a way to recognize last year's awardees’ works, and will open a new stream of advertisement revenue. Another area of discussion is the “impact factor” of the JACMP. It could be argued that the impact factor is of less importance for a clinical journal. However, the clinical faculties at major universities are coming under more pressure to prove their scholarship and for those who publish in the JACMP, a higher impact factor would not hurt. Some techniques used by journals to raise their impact factor are perhaps a little nefarious:Publish higher quality articles at the expense of publishing many fewer submissions. This has the added benefit of lowering costs.Solicit review articles that have high citation impact.JACMP articles with each submission. Some academic journals have such rules and the effect on the impact factor is dramatic.Require authors to cite 3–4 JACMP articles in every submission to other academic journals.Require Section Editors to cite 3–4 Devote, on occasion, an entire issue to a special hot topic(s).A primary goal is to identify a low‐cost application (app) that would allow users of iPhone and Android technologies to view and use the content of the As you see, some of these techniques are ethically a little questionable. Nevertheless, these are actually pretty common publication practices for journals concerned with their impact factor.JACMP has provided approximately 800 open‐access academic articles to the medical physics community without cost of any kind. It is common for the author cost to be in the thousands of dollars to publish open‐access in some other journals. I discussed my ideas on finances within the Editorial in the previous issue. Still, I want to remind all of you that the money to pay for the JACMP needs to come from somewhere. If the money does not come from the author, then where should it come from_ The business model of the JACMP is under active discussion this year.Since its inception, the JACMP receives very little credit respecting academic accomplishment. We need to do what we can to remedy that deficiency.It may be possible to arrange to award continuing education (CE) credits for writing an accepted article or reviewing an article. We should figure out the mechanisms for awarding Self‐Assessment Modules (SAMS) credits for such activities. So much of the work we all do for the JACMP.Some ideas we are considering include our own version of Point–Counterpoint specific to clinical and professional issues, a Book Review Section (we would need a Book Review Editor), and ways to expand the Administrative/Professional Section of the As you see we have our work cut out for us. I will provide a report on our progress in these areas from time to time.Michael D. Mills Editor‐in‐Chief May 27, 2013