Although the peer review system of academic journals is seen as fundamental to scientific achievement, a major threat to the validity of the system is a potential evaluation bias resulting from constraints at the journal level. In this study, we examine how the time pressure to maintain a fixed periodical quota for journal publication can influence a journal editor’s decision to accept or reject a paper at any given point in time. We find that an increase in publication backlog, proxied as the average delay between paper acceptance and print publication, is correlated with an increase in the subsequent rejection rates of new submissions. Our findings suggest that time pressures inherent in the peer review system may be a source of potential evaluator bias, calling for a need to reconsider the current quota system. Modern scientific research blossoms in the form of publications. Today’s peer review process serves as the backbone of modern science, resting on the postulate that publication validates the quality of scientific research –3. IndeeGiven that most academic journals take the form of printed periodicals with fixed paper quotas published at regular time intervals , editors are expected to accept a consistent number of papers for publication at each interval. Editors, who have been called the “gatekeepers of science” , 13, wisIn this study, we examine how such time pressures may have an unintended impact on the editor’s acceptance decisions for submissions. We empirically proxy the number of accepted but not yet published papers using the publication delay between paper acceptance and publication in print. Our analysis provides evidence that when the number of accepted but not yet published papers is sufficient to meet the quota for the next regular issues of the print journal, editors are more likely to reject submissions because the pressure to meet the quota within a given time is low.Time pressure is a ubiquitous phenomenon in our world. Our daily decision making in contexts such as stock trading, shopping, and job searching occurs within limited time constraints that often press us to make quick decisions . A numbeLike those in many contexts, then, editors of academic journals are constantly exposed to time pressure as they must meet their quota for quality papers at regular intervals. If editors fail in this task, they may be viewed by the editorial board, associate editors, and authors as performing poorly, and in some cases, be subject to penalties. Such pressure is likely to be augmented when editors find themselves short of accepted papers for the next regular issues of their journal, a situation in which they may consider accepting papers of less quality that might otherwise be rejected. When editors are subject to less time pressure, however, they may increase their quality threshold for incoming papers and eventually reject even high-quality papers that would have been accepted had they been submitted at a different time.K papers for a journal that needs to be published at calendar time T. The goal of the editor is to maximize the aggregate quality of papers in the journal. At t ∈ , the editor receives a paper with the arrival rate of λ. The distribution of paper quality is known to follow F (q), which does not depend on t. If a paper with the quality q arrives at time t, the editor accepts or rejects the paper based on the given quality of the paper. Let k denote the number of papers which are already accepted. The editor needs to select (k + 1)-th paper for the journal. We are interested in the effect of k, the number of pre-publication accepted papers, on the editor’s time pressure and on the probability of rejection of a received paper, which is a candidate for the (k + 1)-th paper.Based on recent literature on time pressure and task quality, we developed a mathematical model to specify how an editor’s quality threshold for paper acceptance varies according to time pressure, a proxy for which is the backlog of accepted, yet unpublished, papers. The model assumes that an editor seeks to maximize the aggregate quality of papers in a journal according to the constraints of a fixed paper quota while papers of random quality are submitted to the journal at a constant probability. In fact, under such constraints, the greater the number of papers accepted, yet waiting to be published, the lower the possibility for a subsequently submitted paper of equal quality to be accepted. In the formal descriptions of the model that follow, the editor accepts, at most, Proposition 1. When k′ > k, q  ≥ q .q ≥q implies that the probability of acceptance for the (k + 1)-th paper is lower than that of k-th paper because Pr ) ≤ Pr ), which implies that the editor’s quality threshold for manuscript acceptance changes according to time pressure, which is measured by the number of accepted papers. The details for the proof of the proposition is included in our It is worth highlighting that this proposition directly confirms our main argument because per se in a given year, the data instead provides the average publication delay of a given year in months, that is, the average difference in months between the dates of final paper acceptance and publication in print. We used this delay measure as a proxy for the backlog of accepted, but not formally published, papers (these are often referred to as forthcoming or in-press papers and do not have volume and issue information assigned). Then, we empirically examined the relationship between publication delays and rejection rates.We empirically tested our argument using the 2004–2017 annual publication data for 54 journals affiliated with the American Psychological Association (APA) . All thet.In most disciplines only a very small number of manuscripts is accepted for publication on the first submission, and most editors solicit at least one revision from authors. To capture the average review time for APA journals, we randomly selected five papers per publication year from each of APA-affiliated journals, all of which disclosed the specific dates of the first submission and subsequent acceptance for each paper from 2004 through 2017. The average review time was 285 days and the standard deviation was 195.41 days, ranging from a few months to more than two years. Given that the review process period of most of the submitted manuscripts was shorter than two years (less than 730 days), we averaged the publication delays at one and two years prior to the focal year, took the logarithm, and used it as our explanatory variable to see its effect on rejection rate in year t as 1 minus acceptance rate, which is the number of accepted papers in year t divided by the number of received papers in year t. This means that most of the rejections that occur during the review process, which takes up to two years on average, do not immediately factor into the acceptance/rejection rate of the same year. Let us assume that during an evaluation process, an editor rejects a paper submitted in year t-1 or year t-2 based on the publication delay of that year. This decision would, in effect, affect the rejection rate of year t, given that the counterfactual final acceptance would have occurred in year t. Although the 54 APA-affiliated journals in question reported their rejection rates, each academic journal used a different method for calculating the rate. Therefore, for a robustness check, we also tested the extent to which publication delays affect the absolute number of accepted papers. We included two key variables as controls in our analysis. One is the count of items published in print (log) because it indicates the capacity for printed papers of each journal, and the other is the journal impact factor without self citations (log) drawn from Thomson Reuters, since high-quality journal editors may be less concerned about publication delays. Lastly, we included in the model fixed effect dummy variables for journal and year.The reasoning behind this operationalization is that journals usually report their rejection rate in year Prior to the main analysis, we plotted the relationship between rejection rate and publication delay . The leaThe log–log regression coefficient estimates showed that a 1% increase in a journal’s publication delay is correlated with about a 0.1% increase in the journal’s rejection rate, holding other variables constant. We also used the absolute number of accepted papers (log) as our outcome variable. The finding holds with the log of the number of accepted papers as our outcome variable  may influence a journal editor’s time pressure and view of incoming manuscripts. Our finding suggests that even the evaluation of research by an expert academic editor is not completely independent of the time pressure to meet the journal quota. This study has several implications. For authors, publication delays—a proxy for the backlog of accepted but not formally published papers—may work as a factor motivating authors’ choice of journals, given that it influences rejection rates. For editors, we strongly recommend that academic journals publicly disclose their publication delays, or the number of pre-published accepted papers, to help potential authors better time their submissions.empirically test and show the effect of the backlog of accepted papers or publication delays on subsequent rejection rates.A number of researchers have suspected that publication outcome can be affected by the backlog of papers, but there is only one empirical paper , to our More broadly, we believe that it is an opportune time to reconsider the utility of the current quota system of most academic journals. A significant number of journals are still bound by the quota system of print publications despite their concurrent publication of electronic versions of their paper-based journals. We suggest that such a quota-based journal system exposes editors to the pressure of having a constant number of papers ready to print for every regular issue, thus subjecting submitted papers to an evaluation based not only on quality but also on the variable demand for new papers. This problem has been noted by other researchers, who suggest that space shortages (a variable similar to publication delay in this paper) influence journal review processes, though without empirical evidence to support this proposition.Print periodicals have long been touted as one of the most useful forms for sharing academic findings with both researchers and the public on a regular basis. Yet, the cyclical demand for print-based periodicals may lead editors to fall into the selection error of accepting mediocre papers when in need of more content or rejecting quality papers when a sufficient backlog of already accepted papers exists. We contend, therefore, that it is time for editors to have a more flexible approach to page budgeting; rather than limiting the number of issues for a given period they might skip a regular issue when short of high-quality papers or add an issue when a backlog of accepted papers is high.S1 Appendix(PDF)Click here for additional data file. 5 Dec 2019PONE-D-19-25612Does the Pressure to Fill Journal Quotas Bias Evaluation?: Evidence from Publication Delays and Rejection RatesPLOS ONEDear Mr. Park,Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.Two acknowledged experts in the field have commented on your paper. Overall, they find a certain potential in your work to be published, but several key concerns (most of them major), that include biasing sources, interpretations and statistical effects, need your revision and response, as you will see below in the feedback provided by our academic reviewers.https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.We would appreciate receiving your revised manuscript by Jan 19 2020 11:59PM. When you are ready to submit your revision, log on to If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocolsTo enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: Please include the following items when submitting your revised manuscript:A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.We look forward to receiving your revised manuscript.Kind regards,Sergio A. Useche, Ph.D.Academic EditorPLOS ONEJournal Requirements:When submitting your revision, we need you to address these additional requirements:http://www.plosone.org/attachments/PLOSOne_formatting_sample_main_body.pdf and http://www.plosone.org/attachments/PLOSOne_formatting_sample_title_authors_affiliations.pdfPlease ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at  2. Please include a separate caption for each figure in your manuscript.3. We note that you have stated that you will provide repository information for your data at acceptance. Should your manuscript be accepted for publication, we will hold it until you provide the relevant accession numbers or DOIs necessary to access your data. If you wish to make changes to your Data Availability statement, please describe these changes in your cover letter and we will update your Data Availability statement to reflect the information you provide.Reviewers' comments:Reviewer's Responses to QuestionsComments to the Author1. Is the manuscript technically sound, and do the data support the conclusions?The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: PartlyReviewer #2: Partly**********2. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: I Don't KnowReviewer #2: Yes**********3. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: YesReviewer #2: No**********4. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: YesReviewer #2: Yes**********5. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: This was an interesting paper and it seemed correct. However, there is one important thing that the editor should check, and I would be interested to see if other reviewers had this concern as well. Note: I didn’t have time to review the mathematical model, and am mainly provided comments on the empirical data analysis.As I understand it, the data are very coarse, and are measured on an annual basis. It would be much better to have data on a monthly basis, but I understand that might not be available. What would want to try to recover is the editor’s decision at a specific point in time t . Since editors don’t make decisions at one point in time in the year, averaging across the year might generate some bias. I think the bias is probably going to attenuate the effect, but the authors need to more copiously and cogently deal with this issue of measurement error. This is particularly because the effect sizes seem very small to me . It could reflect going from 10% acceptance rate to 9.9%.Further, some of the author decisions seemed sloppy, such as only using one randomly selected paper per year to estimate review time and publication delay. Why could they have not done this with more papers to get a better estimate? Poor measurement of this variable would also probably bias the estimates downward.Reviewer #2: Does the Pressure to Fill Journal Quotas Bias Evaluation? Evidence from Publication Delays and Rejection RatesThis paper sets out to examine how time pressure to maintain a fixed periodical quota for journal publications can influence a journal editor’s decision to accept or reject a paper. The authors find that a 10% increase in publication backlog, is correlated with a 0.8% increase in the subsequent rejection rates of new submissions. This seems very intuitive: when an editor has more papers to choose from, (s)he will reject a higher percentage of submissions. This finding seems to be very much what one would expect. I doubt whether this is a very interesting finding to publish.Subsequently, the authors suggest (p. 9) ‘that a low publication delay period, may decrease the quality of the journal because the editor may assume a more lenient view of newly submitted manuscripts in order to fill the quota.’ Results in Table 7 indeed show that publication delays have a significant positive effect on impact factor. However, more information should be given about the impact factor in general and how it is measured in this paper to determine whether the authors indeed prove what they claim to prove. What  is the relation between time after publication and number of citations? Is this a normal distribution? If so, papers will have a relatively small effect on the impact factor of a journal in the first year after their publication. After how many years/what period will the effect of the quality of papers be felt in the impact factor of a journal? Also, how many years/months before an article was published, was it submitted? It might take quite some time before the quality of an article submitted in year t, published in year t+a, affects the impact factor of the journal in year t+a+b.For this study: from which years were the impact factors taken? How many years difference is there between the year of impact factor and years used to calculate publication delay?The relevance of the study mentioned on page 4, i.e. that a task completed near a deadline is of lower quality than one completed well before the deadline, to the study in this paper is not clear and needs to be explained better. The study mentioned on page 4 is about the effect of time pressure on the work someone is performing him/herself. The paper under review is about the effect of time pressure felt by the editor on the quality of accepted work done by others than the editor.Figures 2 and 3: it is not clear to me how to read these figures. This has to be explained better.**********what does this mean?). If published, this will include your full peer review and any attached files.6. PLOS authors have the option to publish the peer review history of their article :Reviewers' comments:Reviewer's Responses to QuestionsComments to the Author1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.Reviewer #1: All comments have been addressed**********2. Is the manuscript technically sound, and do the data support the conclusions?The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented. Reviewer #1: Yes**********3. Has the statistical analysis been performed appropriately and rigorously? Reviewer #1: Yes**********4. Have the authors made all data underlying the findings in their manuscript fully available?PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.The Reviewer #1: Yes**********5. Is the manuscript presented in an intelligible fashion and written in standard English?PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.Reviewer #1: Yes**********6. Review Comments to the AuthorPlease use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. Reviewer #1: This was an impressive revision. The authors conducted several additional analyses and data collection efforts to address my concerns. I assumed the editor believed my initial comments could be handled.**********what does this mean?). If published, this will include your full peer review and any attached files.7. PLOS authors have the option to publish the peer review history of their article (If you choose “no”, your identity will remain anonymous but your review may still be made public.Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.Reviewer #1: No 30 Jul 2020PONE-D-19-25612R1 Does the Pressure to Fill Journal Quotas Bias Evaluation?: Evidence from Publication Delays and Rejection Rates Dear Dr. Park:I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department. onepress@plos.org.If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact plosone@plos.org. If we can help with anything else, please email us at Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staffon behalf ofDr. Sergio A. Useche Academic EditorPLOS ONE


A crucial, yet subjective and non-evidence-based, decision for researchers is where to submit their original research manuscripts. The approach of submitting to journals in descending order of impact factor (IF) is a common but imperfect strategy. The validity of the IF as a measure of journal quality and significance is suspect, and a number of other journal impact scores have emerged, such that no one scale is universally accepted. Furthermore, practical considerations, such as likelihood of manuscript acceptance rates and times for decisions, may influence how authors prioritize journals. In this report, we sought to 1) review emergency medicine (EM) journal impact metrics, and 2) provide a comprehensive list of pertinent journal characteristics that may influence researchers’ choice of submission.We systematically reviewed five impact metrics  and other relevant characteristics of 20 EM journals.We found good to excellent agreement in ordinal rankings of four of the journal impact metrics, as measured by the Spearman rank correlation coefficient. The median acceptance rate for original research manuscripts in the EM category was 25% , and the median initial decision time was 33 days . Fourteen EM journals (70%) accepted brief reports, and 15 (75%) accepted case reports/images.We recommend replication, expansion, and formalization of this repository of information for EM investigators in a continuously updated, open-access forum sponsored by an independent organization. Many experts have questioned the validity of the IF as a measure of journal quality and influence.5With over 5200 journals currently indexed in Medline,Beyond the limitations of relying on one or more impact metrics, researchers must consider the likelihood of acceptance, time until decision, reach of audience, and expected number of citations. Although the comments to authors after rejections may help improve subsequent submissions, reflexively submitting to high-prestige journals with low likelihood of acceptance can nevertheless waste inordinate amounts of time for decisions and effort toward serial reformatting for particular journal requirements.With minimal published guidelines, a common approach for junior (and other) investigators seeking assistance in manuscript submission decisions is to turn to senior academicians for advice – ironically rendering this critical step in their otherwise objective scientific work into a subjective, non-evidence based process. The single, objectively derived decision model for manuscript submissions is one proposed by Wong et al, which requires multiple inputs including journals’ manuscript acceptance rates, times for first decision, and open access fees that may not be readily available.After review of the most commonly used journal impact metrics,As a secondary analysis, we sought to compare the agreement of the ordinal rankings of the five IF metrics, ie, the correlation between how the individual metrics ranked journals. For this analysis, we calculated the Spearman rank correlation coefficient, rho, for each pairwise combination of metrics. We conducted these analyses using Stata v13  and Excel X for Mac .With an explicit goal to provide practical, readily available information to inform submission decisions, we reviewed literature  about pertinent journal characteristics,What is your acceptance rate for original research manuscripts (# accepted for publication/# submitted)?What is your median or mean time for decisions on submitted manuscripts (how many days/weeks/months from submission to time that a decision is rendered and sent to the authors)?We sent four follow-up emails to non-responders at 10-day intervals and a final inquiry a month after the fourth request. When journals provided vague or incomplete information, the lead investigator asked for further clarification from their editorial staff. As a review of published materials without any patient considerations, this project was categorized as exempt from institutional review board review.Current Heart Failure Reports, MicroRNA, and journals that did not typically publish original research manuscripts (Emergency Medicine Clinics of North America). We also excluded journals that did not have a 2017 IF on the 2018 Journal Citation Reports 2018 Clarivate Analytics report of IFsTo generate the journal list, we reviewed the list of top 30 journals in the EM category on the Scimago Journal & Country Rank website .Of the 30 journals listed in the EM category, we excluded eight for irrelevant or narrow focus, one for not publishing original research, and one because of no IF in 2017. We present impact factor metrics and other characteristics of the remaining 20 EM journals in We present the ranking of EM journals by summation of impact factor metrics in Original research investigations are generally laborious and lengthy, often consuming years from start to finish. When considering where to submit the final product of their research for publication, investigators should be afforded as much objective, easily accessible information as possible. Toward this end, we sought to provide a comprehensive review of EM journal impact metrics and other characteristics for investigators.We found that all but one of the impact metrics showed good to excellent agreement in their ordinal rankings, suggesting that these metrics and their formulas capture only slight nuances in impact. The poor correlation of the H index may be due to the fact that it is generally intended as a metric for authors – not journals. Although several websites provide general descriptions of these and other impact metrics, we were unable to find a similar specific analysis of journal impact metric correlation in any subspecialty field of medicine.We are not advocating that our summary impact ranking is a general proxy of journal quality, and it should not become a de facto “one-two-three…” template for sequential targeted submission. Detailing all the factors that influence journal choices is beyond the scope of this work. The journal characteristics and the criteria for journal inclusion on these lists were chosen by a single investigator after review of the literature and consideration of the submission decision model proposed by Wong et al. EM investigators and their research are eclectic, and their publication priorities reflect this breadth of experience.Although the information presented in this study is purported to be readily available, we were surprised by the difficult and time-consuming nature of the data collection process. We anticipated that we would only need to conduct simple searches over a month (or less) to gather our desired data – it took nearly five months. Although three websites provided much of the standardized journal impact metric data,Very few journals published information regarding acceptance rates and decision times, and only 28% of our first email inquiries to journal staff were answered. Given these difficulties, we recommend the development of an independently maintained, expanded repository that gathers this information on an ongoing basis – with our work and tables providing a template or roadmap toward this goal. Considerations of conflict of interest or bias toward their affiliated journals notwithstanding, the most logical sponsors of such an endeavor are EM professional organizations such as the Society for Academic Emergency Medicine or the American College of Emergency Physicians. Regardless of who performs this service, the most appropriate home for its output is a freely available, open access website. From a sustainability standpoint, we expect that journals would, over time, recognize the benefits of collaboration and transparency of this repository and provide the input data more freely.all types of manuscripts. Similarly, in terms of median/mean times for decisions, their data may have been skewed if they referred to all submissions, including those that were immediately rejected and not sent out for review. Furthermore, use of the median and mean decision times without standard deviations or IQRs of the individual journals may obscure another important factor – the variation in time to decisions within a journal.While journal impact metrics are calculated by third parties in an objective, standardized fashion, the primary limitation of this report is the reliance on journal self-reports for acceptance rates and times for decisions. A few journals either did not respond to our inquiries or stated that they do not provide this information, and so this data remains incomplete. Furthermore, even though we specifically requested data regarding original research, some journals may have provided acceptance rates for We present summary tables of EM journal impact metrics and characteristics to inform original research manuscript submission choices for EM investigators. Considering the effort to acquire this data and annual changes in journal impact metrics, we recommend development of a centralized, open access website repository that can be updated from year to year.


Background: "Open science" is an umbrella term describing various aspects of transparent and open science practices. The adoption of practices at different levels of the scientific process  has been rapidly changing the scientific research landscape in the past years, but their uptake differs from discipline to discipline. Here, we asked to what extent journals in the field of sleep research and chronobiology encourage or even require following transparent and open science principles in their author guidelines.Methods: We scored the author guidelines of a comprehensive set of 27 sleep and chronobiology journals, including the major outlets in the field, using the standardised Transparency and Openness (TOP) Factor. The TOP Factor is a quantitative summary of the extent to which journals encourage or require following various aspects of open science, including data citation, data transparency, analysis code transparency, materials transparency, design and analysis guidelines, study pre-registration, analysis plan pre-registration, replication, registered reports, and the use of open science badges.Results: Across the 27 journals, we find low values on the TOP Factor  in sleep research and chronobiology journals.Conclusions: Our findings suggest an opportunity for sleep research and chronobiology journals to further support recent developments in transparent and open science by implementing transparency and openness principles in their author guidelines. The term “open science” is relatively ill-defined and includes a range of different methods, tools, platforms, and practices that are geared to improving the quality of science through transparency3. At present, it is still largely up to individual researchers and research groups to decide to what extent they want to engage in open science practices and incentives that may promote open science are rare. Journals as the main outlets for archival scientific dissemination can support the movement and offer ways to make the scientific process more open, reproducible, and emphasise good scientific practice. They may even speed up the process by requiring authors to adhere to open science standards. However, to what extent do journals in the fields of sleep and chronobiology encourage or even require following the standards of open science?During the past few years, the open science movement gained increasing popularity and is rapidly changing the way science is done, especially among early career researchers striving to improve scientific practice and overcome deficits in the current scientificThe scientific fields of sleep research and chronobiology concern all aspects of sleep and circadian rhythmicity. As almost all aspects of physiology and behaviour are under some type of circadian control, this cluster of scientific fields is fundamentally interdisciplinary, employing a wide variety of methodologies. Therefore, this research area is very heterogeneous, drawing from different ‘core’ disciplines , each with their own scientific history, and the degree to which open science principles are adopted may vary widely.5, we assessed the implementation of research transparency and openness in journal guidelines using the quantitativeTransparency and Openness Factor6 (TOP Factor). The TOP Factor, launched in February 2020, is based on the TOP guidelines and provides a normative standard for research transparency and openness. It contains ten sub-scales, corresponding to different aspects of openness and transparency in scientific research, reflecting theTransparency and Openness Promotion Guidelines: data citation, data transparency, analysis code transparency, materials transparency, design and analysis guidelines, study pre-registration, analysis plan pre-registration, replication, registered reports, and open science badges.In this study, we asked to what extent scientific journals specialised on sleep research and chronobiology lay out open-science principles in their author guidelines. Inspired by previous publications in the field of pain researchThe TOP Factor recognises different levels relating to mentioning, encouraging, requiring and enforcing specific transparency and openness practices, which are implemented in a verbally anchored rating scheme. The specific practices are:Data citation refers to the citation of data in a repository using standard means, including a digital object identifier (DOI). Citing the data set(s) that is/are reported on in a given publication facilitates access to these data, independent re-analyses by other authors times, and validation of results, thereby likely increasing the probability that a given result can be reproduced by a different group, or at a later time.Data, analysis code, andmaterials transparency refers to making data, analysis code and materials available as part of the journal submission. The availability of original data again supports the reproducibility of a given reported research result, as other investigators will be able to analyse the data independently. Furthermore, the availability of data also supports meta-analysis or evidence synthesis efforts. The availability of code is a necessary step to be able to reproduce the exact same analysis again at a later time. Finally, availability of materials used in the study ensures that other investigators can independently replicate the research design and expand on it in a way that is consistent with the original publication.Design and analysis guidelines refers to the inclusion of instruments describing the study design and analysis formally, such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) or Consolidated Standards of Reporting Trials (CONSORT) standards. These standard methods aid in providing an overview of the methodological rigour of the study and facilitate appraisal of completeness of a given research study.Study pre-registration andanalysis pre-registration refers to the pre-registration of data collection and/or analysis prior to their execution. Pre-registering study designs and analysis can be a countermeasure against HARKing (hypothesizing after the results are known)7 and other questionable research practices (QRPs)8. Pre-registration can take many forms, and most often refers to specifying the research design in advance of data collection, as well as the data analysis plan, so as to avoid “fishing” for positive results. Replication refers to an explicit desire of the journal to include articles not based on novelty. This reflects the implicit mission of some journals to publish only novel journals, and not simply replications of previous work. Publishing replications of previous work and their results to some degree supports independent confirmation of the previous results, and also to a certain extent, addresses the file drawer problem9.Registered reports refers to prospective peer review, i.e. evaluation of a manuscript submitted to a journal prior to data collection and/or data analysis. Registered reports have recently gained significant traction, with a few high-profile journals, including Nature Human Behaviour, Nature Communications, Scientific Reports and PLOS Biology, accepting this article that ‘frontloads’ peer review.The categoryOpen-science badges refers to the use of so-called badges, which are awarded if a paper adheres to specific standards, thereby providing an incentive for promoting transparency and openness10.In summary, the TOP Factor covers major dimensions of open science and provides a helpful and standardised tool that allows to compare between journals or fields the extent to which they encourage or require adherence to open science principles.11) is a quantitative score summarising the presence, requirement, and enforcement of transparent and open science practices in journals. It includes a total of ten sub-scales, of which nine score 0–3, and one scores 0–2, thereby resulting in a maximal summed score of 29. Higher values indicate a higher degree of adherence to the TOP practices.The TOP Factor . The search terms, entered in separate searches, were:“sleep”“chronobiology”“circadian”“biological rhythms”“dream”Secondary strategies. In addition to the primary search strategy, we used two supportive secondary strategies to identify relevant journals that may have been missed in the primary strategy:“Own domain-relevant expertise in sleep and chronobiology;“Informal consultation with a senior researcher with >25 years of experience in the field.Validation. The search results were merged, and duplicates were removed. We validated our search strategy by confirming that all journals listed in a recent publication on sleep research journals13 were identified using this strategy. We validated our search strategy by confirming that the above search terms produce the same list of journals in MEDLINE .https://www.cos.io/initiatives/top-guidelines) for “sleep”, “chronobiology”, “circadian”, “biological rhythms” and “dream”, and one through a search in theNational Library of Medicine (NLM).In addition to this strategy, we found two additional journals via searching the list of TOP signatories  and the 5-year Journal Impact Factor from theInternet Wayback Machine.We consulted the journal websites for author guidelines. Where possible, we archived journal guidelines either locally, or on the11). We then discussed and resolved major sources of discrepancy , resolved some per-item disagreements and rescored the categories “Data citation” , “Reporting guidelines”  and “Study pre-registration”  independently in a second pass (see underlying data11). We report Fleiss’ kappa14, intraclass correlation , and the disagreement rate for each item before resolving discrepancies. At the end of this second pass, we again entered a consensus process, by the end of which all ratings agreed. All scoring was completed between mid-May and mid-June 2020. Results are reported inThree scorers  independently assessed the 27 identified journals’ TOP Factors in a total of 270 individual ratings . In a first pass, the three scorers agreed in 77.7% of all ratings , and the 5-year JIF was available for all of these 15 except one . Out of these 28 journals 11 out of 28, 39.2% of journals were not at present supported by a society. Three journals accepted submissions in a language other than English. Through our hybrid search strategy, we also found six additional journals , which we did not further consider, as they did not focus on sleep research or chronobiology, or were general purpose journals. One journal,th percentile 1, 75th percentile 4, minimum 0, maximum 9, IQR 3) out of a maximum of 29 points (Clocks & Sleep (9),Sleep Science and Practice (7), andSleep and Vigilance (6). Interestingly, these three journals were founded no earlier than 2017.Across the 27 journals we examined, we find a total median TOP Factor of 3  or 5-year IF and TOP Factor .To determine if journal prestige is correlated with TOP factor, we examined the correlation between IF and 5-year IF with Top Factor, finding no evidence for a significant rank correlation between IF and TOP Factor , the median TOP Factor is 2 . Our subfield data are also comparable to a recent study examining transparency and openness in the field of pain research, which found a median TOP Factor of 3.5 (IQR 2.8)5.Our results, which focus on sleep research and chronobiology journals exclusively, are comparable to data on the uptake of the TOP guidelines across other disciplines. In 412 journals included in the Center for Open Science data base , Level 1 is satisfied if the research was registered in an independent, institutional registry, specifying “study design, variables, and treatment conditions prior to conducting the research”, leaving the level of detail open and rendering scorings ambiguous.There can be large ambiguity in the extent that a journal implements specific transparency and openness standards. Take, for example, the category “Study pre-registration”. There are four levels in this category: Level 0:registration vs.pre-registration16. While the registration of a clinical trial in a trial registry such as clinicaltrials.gov can be relatively lightweight and contain only minimal details, a pre-registration (as used in the open science community) typically refers to the prospective specification of concrete study details, including methodology, sample size, and analysis plan prior to data collection in registries such asAsPredicted.org (https://www.aspredicted.org/) orOSF Preregistration (https://osf.io/prereg/)17.Indeed, there is a debate and confusion regarding the use of the terms18), they have to provide a short description of the trial, name the study goals, describe the intervention, name the primary endpoints, inclusion and exclusion criteria, the final sample size , and the sponsor. Clearly, although the degree of detail is of course also subject to variation among pre-registered studies, the required level of detail for registering clinical studies is rather low, with accountability consequently likewise being very low. In some legislations (such as Switzerland), the submission of ethics application as a clinical trial (which is required for some studies that modify sleep schedules), by default deposits the study in the (Swiss) clinical trial registry19.In more detail, the registration of a clinical trial in a registry such as clinicaltrials.gov on the one hand, and the pre-registration of analysis procedures and hypotheses prior to conducting the research on the other hand, mostly serve fundamentally different purposes, which is reflected in their nature too. First, clinical studies, which have not been registered, are impossible to publish in respected journals rendering the process a necessity rather than a self-imposed step to improve scientific transparency. Generally, when authors register a clinical trial . Publishing the review report and disclosing the reviewer identity are independent aspects of the broader ‘open review’ concept, as it is possible to disclose the reviewer identity to the authors as part of the journal submission and publication process without publishing it as part of the article, and it is also possible to publish the review report in anonymised form. There may of course be negative effects of disclosing the identity of reviewers, and potential reviewers may self-censor or fear retaliation if their names are known. Publishing the review reports does not only make the journey of an article from submission to publication transparent, laying open the shaping of an article through the peer review process. It can also curtail unreasonable requests during peer review and may encourage reviewers to provide constructive feedback oriented towards the best scientific outcome. We therefore encourage to include21. As principles of open science are becoming more important in hiring and funding decisions, the TOP Factor may similarly be subject to being ‘gamed’, with ‘openwashing’, i.e. the possible practice of superficially claiming adherence to open-science principles without ‘true’ engagement with them, being an immediate concern22. Ultimately, whether or not this is a key driver will only be available for study retrospectively.Quantitative metrics can be subjected to being ‘gamed’, i.e., manipulated in such a way that they no longer correspond to corresponding to the original objective of the metricWhile the TOP Factor represents a first an important step in understanding the uptake of open and reproducible science systematically, it is important to note that some researchers may not be able to engage with open and reproducible science, due to institutional requirements, not least due to the effort required to implement them. Furthermore, there may be privacy or intellectual property concerns that could prevent the release of data, or the release of code pipelines. It is also conceivable that journals may not see open and reproducible practices as ‘nice to have’ but not ‘must have’ features or may think that they place a burden on the authors, thereby making the journal less attractive. This highlights the complexity of translating concepts of open, transparent and reproducible science into practice.23. The TOP Factor represents but one way of considering the extent to which one aspect of the research ecosystem – journals – can be summarised.We further want to highlight the need to consider open science not as one ‘movement’, but as a set of practices, principles and policies from a wide and diverse “buffet” of optionsIn a comprehensive analysis of the author guidelines for 27 sleep research and chronobiology journals, we have found low evidence for explicit implementation of open and transparent science principles as assessed by the TOP Factor. We therefore encourage journals to make their requirements more explicit. Furthermore, to promote the recent developments in open, transparent and reproducible, journals should provide incentives for following open science practices.http://www.doi.org/10.17605/OSF.IO/KTMBHOpen Science Framework: Transparency and open science principles in reporting guidelines in sleep research and chronobiology journals – Underlying and extended data.This project contains the following underlying data:- SupplementaryInformation_S1.xlsx - SupplementaryInformation_S2.xlsx http://www.doi.org/10.17605/OSF.IO/KTMBH this project contains the following extended dataOpen Science Framework: Transparency and open science principles in reporting guidelines in sleep research and chronobiology journals – Underlying and extended data.- SupplementaryInformation_S3.pdf (TOP Factor scoring rubric)Creative Commons Attribution 4.0 International license (CC-BY 4.0).Data are available under the terms of thehttp://www.doi.org/10.1101/2020.06.26.17294024A previous version of this article is available from bioRxiv: I have no further comments.Is the work clearly and accurately presented and does it cite the current literature?YesIf applicable, is the statistical analysis and its interpretation appropriate?YesAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?YesAre the conclusions drawn adequately supported by the results?YesAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:Sleep and MemoryWe confirm that we have read this submission and believe that we have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. I have no further suggestions or comments.  I recommend publication.Is the work clearly and accurately presented and does it cite the current literature?YesIf applicable, is the statistical analysis and its interpretation appropriate?Not applicableAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?YesAre the conclusions drawn adequately supported by the results?YesAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:NAI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. It is important to describe the extent to which journals support transparency so that we can help journals support greater transparency and openness through their policies and procedures.Replication refers to an explicit desire of the journal to include articles not based on novelty,” which is part of, but not the entirety of, a TOP standard designed to promote the publication of replication studies. It is unclear how the authors handled this and other compound standards/questions. The discussion also raises the problem of unclear/imprecise language like “should,” but the manuscript does not explain how the authors coded policies with unclear language. The TOP Factor is an appropriate framework to describe these issues, but it has some limitations. Notably, I would not describe TOP factor as an “instrument.” Like the TOP guidelines, the TOP factor includes normative standards for transparency. It does not provide structured questions to determine whether those standards have been met. Some aspects of the TOP guidelines are difficult to rate because the standards were not developed to be used as a rating instrument. Each of the TOP guidelines includes multiple components, and journal policy language can be unclear and contradictory. For example, this manuscript says that “ The methods used to assess agreement are very limited. Given the proportion of journal policies that were rated level 0, the total agreement should be extremely high. For example, it is concerning that the authors disagreed about 13/28 journals for one item, which suggests that the “instrument” is not reliable. At a minimum, the authors should report kappa and specific agreement , and they should consider the implications of poor reliability for efforts to measure and improve transparency and openness. The authors rated a small number of journals related to sleep research, a field in which I am not an expert. It seems possible that these journals would be representative of this specific field of research. Transparency is poor in many disciplines, but these specific results might have limited generalizability as well as limited internal validity.Is the work clearly and accurately presented and does it cite the current literature?YesIf applicable, is the statistical analysis and its interpretation appropriate?PartlyAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?PartlyAre the conclusions drawn adequately supported by the results?PartlyAre sufficient details of methods and analysis provided to allow replication by others?NoReviewer Expertise:Clinical trials, evidence synthesis, research transparencyI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. We thank the author for their comments, and agree that there is probably room for improved in the methodology that we used. We have removed the description of the TOP Factor as an instrument in the abstract. We have also now reported kappa, ICC, and % agreement.  Overall This is an interesting article on an important topic. There is not sufficient discussion or rationale for some of the statements relating to the importance of this topic. The authors assume that transparency and open science principles are already accepted by all scientists/journal and that all aspects of these principles are equally important. More text about why these principles are important overall and for each principle as well as a recognition why a scientist or journal might not want to adhere to all principles would make this a more balanced report. Minor comment: some text is in the wrong area .SpecificAbstract: need more justification for why adherence should be mandatory.rd paragraph:P2 Introduction, 3 Refs 4 and 5 are both in the same field of pain; that is not evidence of “publications in other fields” .P3 Introduction: Please add text about why each principle or group of principles are important for science.P3 Methods: Reference 16 is to another version of this paper. It is not independent validation.P3 Methods; bullet beginning “The search results were merged…”: The Validation text should be under Validation bullet.P4 Method, left panel: What strategy was used to find the two additional journals?P4 Methods. Journal meta-data extraction: Information about what percent of journal were supported by a society and what percent were in language that was not English belongs in Results (unless those criteria are exclusionary).P4 Methods. Journal guidelines extraction: Information about no public author guidelines belongs in Results.P4 Methods:The date of registration of this study in a registry and the name of the registry is not given. When I looked at the registry (from a link in this paper), this study was registered on July 13, after the research had begun. Is that correct? Did the authors not follow TOP principles in this paper?P4 Results: Begin with the number of journals found.P4 Results. Last line of left panel: The text beginning “our Results compare…” belongs in Discussion.P4 Results “Lack of a standard specification for journal guidelines”: It is not clear why the information in the first two sentences should be considered a problem. Most of the rest of the paragraph belongs in the Discussion.P4 Discussion: Please give a few examples of such registries in the main text or in Supplemental materials. Readers might not be aware of the many ones available.P9 Discussion: Another reason besides laziness why are researcher might not want to complete optional requirements is the many other tasks they might need to complete that are not optional.P9 Discussion: Please acknowledge the potential negative effect of having the name of a reviewer revealed. I personally feel like I can be less honest when my name is known to the authors because of potential effects on our inter-personal relationship(s). I don’t think I am unreasonable in my requests – and the author can always refuse to do those requests.P9 Discussion: Please also discuss/acknowledge why a researcher or journal might not want to fully comply with all TOP factors.S1 and S1: What were the criteria for excluding the journals on the Removed tab?S3 PDF: Please label what it is within the document.Is the work clearly and accurately presented and does it cite the current literature?YesIf applicable, is the statistical analysis and its interpretation appropriate?Not applicableAre all the source data underlying the results available to ensure full reproducibility?YesIs the study design appropriate and is the work technically sound?YesAre the conclusions drawn adequately supported by the results?YesAre sufficient details of methods and analysis provided to allow replication by others?YesReviewer Expertise:sleep research and chronobiologyI confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Reviewer 2This is an interesting article on an important topic. There is not sufficient discussion or rationale for some of the statements relating to the importance of this topic. The authors assume that transparency and open science principles are already accepted by all scientists/journal and that all aspects of these principles are equally important. More text about why these principles are important overall and for each principle as well as a recognition why a scientist or journal might not want to adhere to all principles would make this a more balanced report. Minor comment: some text is in the wrong area .SpecificAbstract: need more justification for why adherence should be mandatory.  rd paragraph: Refs 4 and 5 are both in the same field of pain; that is not evidence of “publications in other fields” .P2 Introduction, 3  P3 Introduction: Please add text about why each principle or group of principles are important for science.  P3 Methods: Reference 16 is to another version of this paper. It is not independent validation.  P3 Methods; bullet beginning “The search results were merged…”: The Validation text should be under Validation bullet.  P4 Method, left panel: What strategy was used to find the two additional journals?  P4 Methods. Journal meta-data extraction: Information about what percent of journal were supported by a society and what percent were in language that was not English belongs in Results (unless those criteria are exclusionary).  P4 Methods. Journal guidelines extraction: Information about no public author guidelines belongs in Results.  P4 Methods:The date of registration of this study in a registry and the name of the registry is not given. When I looked at the registry (from a link in this paper), this study was registered on July 13, after the research had begun. Is that correct? Did the authors not follow TOP principles in this paper?  P4 Results: Begin with the number of journals found.  P4 Results. Last line of left panel: The text beginning “our Results compare…” belongs in Discussion.  P4 Results “Lack of a standard specification for journal guidelines”: It is not clear why the information in the first two sentences should be considered a problem. Most of the rest of the paragraph belongs in the Discussion.  P4 Discussion: Please give a few examples of such registries in the main text or in Supplemental materials. Readers might not be aware of the many ones available.  P9 Discussion: Another reason besides laziness why are researcher might not want to complete optional requirements is the many other tasks they might need to complete that are not optional.  P9 Discussion: Please acknowledge the potential negative effect of having the name of a reviewer revealed. I personally feel like I can be less honest when my name is known to the authors because of potential effects on our inter-personal relationship(s). I don’t think I am unreasonable in my requests – and the author can always refuse to do those requests.  P9 Discussion: Please also discuss/acknowledge why a researcher or journal might not want to fully comply with all TOP factors.  S1 and S1: What were the criteria for excluding the journals on the Removed tab?  S3 PDF: Please label what it is within the document. We thank the reviewer for the comments.Abstract: need more justification for why adherence should be mandatory. We have now revised the abstract, and have removed the reference to mandatory adherence. Ultimately, we do not think that forcing authors to adhere to specific guidelines will lead to culture change.P2 Introduction, 3rd paragraph: Refs 4 and 5 are both in the same field of pain; that is not evidence of “publications in other fields” . We have now revised this sentence to specifically refer to pain research.P3 Introduction: Please add text about why each principle or group of principles are important for science. We now added more text describing why each of these principles may be important for science, giving the reader more context.P3 Methods: Reference 16 is to another version of this paper. It is not independent validation.Journal of Sleep Research , which examined the Impact Factor of sleep research journals. We now refer to Reference 10, a publication in theP3 Methods; bullet beginning “The search results were merged…”: The Validation text should be under Validation bullet. We have now moved this to the Validation section.P4 Method, left panel: What strategy was used to find the two additional journals? We have now included this information. P4 Methods. Journal meta-data extraction: Information about what percent of journal were supported by a society and what percent were in language that was not English belongs in Results (unless those criteria are exclusionary). We have now moved this to the Results section.P4 Methods. Journal guidelines extraction: Information about no public author guidelines belongs in Results.  We have now moved this to the Results section.P4 Methods:The date of registration of this study in a registry and the name of the registry is not given. When I looked at the registry (from a link in this paper), this study was registered on July 13, after the research had begun. Is that correct? Did the authors not follow TOP principles in this paper?Extended Dataset, which we uploaded to the Open Science Framework with a registration date of 13 July 2020 (https://doi.org/10.17605/OSF.IO/KTMBH), in line with recommendations from Wellcome Open Research. Our pre-registration of the work was registered on 14 May 2020 (https://doi.org/10.17605/OSF.IO/QNSBM). The authors must have looked at theP4 Results: Begin with the number of journals found. We have now included this in the Results section.P4 Results. Last line of left panel: The text beginning “our Results compare…” belongs in Discussion. We have now moved this to the Discussion section.P4 Results “Lack of a standard specification for journal guidelines”: It is not clear why the information in the first two sentences should be considered a problem. Most of the rest of the paragraph belongs in the Discussion. This paragraph was moved to the discussion.P4 Discussion: Please give a few examples of such registries in the main text or in Supplemental materials. Readers might not be aware of the many ones available. We have now added references to the two main pre-registration sites: AsPredicted.org and OSF Pregistration.P9 Discussion: Another reason besides laziness why are researcher might not want to complete optional requirements is the many other tasks they might need to complete that are not optional. We have revised this to include the nuance that researchers may simply prioritise non-optional tasks.P9 Discussion: Please acknowledge the potential negative effect of having the name of a reviewer revealed. I personally feel like I can be less honest when my name is known to the authors because of potential effects on our inter-personal relationship(s). I don’t think I am unreasonable in my requests – and the author can always refuse to do those requests. We have now revised this section to say:Open review as an additional category in future developments of the TOP guidelines, including the previously described shape that open peer review can take. Some journals, including eLife, PLOS, and Clocks & Sleep, now offer publishing of the pre-publication peer-review, with the possibility of naming the reviewers (if they agree). Publishing the review report and disclosing the reviewer identity are independent aspects of the broader 'open review' concept, as it is possible to disclose the reviewer identity to the authors as part of the journal submission and publication process without publishing it as part of the article, and it is also possible to publish the review report in anonymised form. There may of course be negative effects of disclosing the identity of reviewers, and potential reviewers may self-censor or fear retaliation if their names are known. Publishing the review reports does not only make the journey of an article from submission to publication transparent, laying open the shaping of an article through the peer review process. It can also curtail unreasonable requests during peer review and may encourage reviewers to provide constructive feedback oriented towards the best scientific outcome. We therefore encourage to includeP9 Discussion: Please also discuss/acknowledge why a researcher or journal might not want to fully comply with all TOP factors. We now address this in the discussion: S1 and S1: What were the criteria for excluding the journals on the Removed tab? We have now stated these criteria in the Results section.S3 PDF: Please label what it is within the document.Extended Data section. This is included in the The authors assess the implementation of open science practices in journal author guidelines in the field of sleep and chronobiology. The authors achieved this by calculating the Transparency and Openess (TOP) factor for 28 sleep and chronobiology journals. They find that overall, open science practices are poorly represented within sleep and chronobiology journals and call for more explicit open science guidelines, e.g. via replacing “soft encouragement” with “hard requirements”. The implementation of open science practices is an important matter and the manuscript helps to raise awareness that many publishers still do not encourage or require these practices. With that said, the present work only provides a limited overview for the field of sleep and chronobiology, without providing a goal post or a broader reference point (except for data from the field of pain research). I think a broader comparison across several fields would be more informative. However, given that few papers acknowledge open science issues in sleep research this paper is a breath of fresh air towards important discussion of such matters.What I am missing is a reference point. The authors mention the TOP factor in pain research on page 4, but I would like to see some general information about the distribution of TOP factors and a descriptive comparison of where sleep journals stand relative to others in general, as e.g. done in this blog article:https://www.natureindex.com/news-blog/top-factor-rates-journals-on-transparency-openness. After all, sleep research is not only published in sleep-related journals.Maybe explicitly mention when the TOP factor was launched (Feb 2020), i.e. how much time there was to implement it. Of course, open science practices could (and should) have been implemented before the TOP factor, but given that the measure is around for a short period of time, it is reasonable to expect that journal requirements are not yet geared towards it.https://go.nature.com/3gat5Vl). It may also be worth discussing whether the TOP factor is enough to encourage good science. Especially in sleep science there is a tendency to collect very small samples and as far as I can see there is no measurement for this included in the TOP factor . Also, funders and universities may play a much more important role in incentivizing open and reproducible science.Another point to discuss would be the potential misuse of the TOP factor, i.e. some form of “gamification” or “open washing”, where the TOP factor becomes the new impact factor  might be too small for a meaningful overview. Then it would be useful to nevertheless offer a scatter plot.https://bit.ly/2X0nOrJ).The authors rightly point out that journals should adopt hard requirements to encourage scientists to practice open science (page 9). This is then followed by a series of reasons why researchers have not adopted open science practices e.g., it costs time. It might be nice to remind the reader of the positive reasons to adopt open science practices  factor for 28 sleep and chronobiology journals. They find that overall, open science practices are poorly represented within sleep and chronobiology journals and call for more explicit open science guidelines, e.g. via replacing “soft encouragement” with “hard requirements”. The implementation of open science practices is an important matter and the manuscript helps to raise awareness that many publishers still do not encourage or require these practices. With that said, the present work only provides a limited overview for the field of sleep and chronobiology, without providing a goal post or a broader reference point (except for data from the field of pain research). I think a broader comparison across several fields would be more informative. However, given that few papers acknowledge open science issues in sleep research this paper is a breath of fresh air towards important discussion of such matters.What I am missing is a reference point. The authors mention the TOP factor in pain research on page 4, but I would like to see some general information about the distribution of TOP factors and a descriptive comparison of where sleep journals stand relative to others in general, as e.g. done in this blog article:https://www.natureindex.com/news-blog/top-factor-rates-journals-on-transparency-openness. After all, sleep research is not only published in sleep-related journals.Maybe explicitly mention when the TOP factor was launched (Feb 2020), i.e. how much time there was to implement it. Of course, open science practices could (and should) have been implemented before the TOP factor, but given that the measure is around for a short period of time, it is reasonable to expect that journal requirements are not yet geared towards it.https://go.nature.com/3gat5Vl). It may also be worth discussing whether the TOP factor is enough to encourage good science. Especially in sleep science there is a tendency to collect very small samples and as far as I can see there is no measurement for this included in the TOP factor . Also, funders and universities may play a much more important role in incentivizing open and reproducible science.Another point to discuss would be the potential misuse of the TOP factor, i.e. some form of “gamification” or “open washing”, where the TOP factor becomes the new impact factor  might be too small for a meaningful overview. Then it would be useful to nevertheless offer a scatter plot.https://bit.ly/2X0nOrJ).The authors rightly point out that journals should adopt hard requirements to encourage scientists to practice open science (page 9). This is then followed by a series of reasons why researchers have not adopted open science practices e.g., it costs time. It might be nice to remind the reader of the positive reasons to adopt open science practices , the median TOP Factor is 2 .  Our subfield data are also comparable to a recent study examining transparency and openness in the field of pain research, which found a median TOP Factor of 3.5 (IQR 2.8) This now serves as a useful reference for the comparison, indicating that the results found here are very much comparable with other journals as well.Maybe explicitly mention when the TOP factor was launched (Feb 2020), i.e. how much time there was to implement it. Of course, open science practices could (and should) have been implemented before the TOP factor, but given that the measure is around for a short period of time, it is reasonable to expect that journal requirements are not yet geared towards it. Point 2: We have now addressed this in the introduction, stating that it was launched in February 2020, and also have added a section about this in the discussion. We write: As the TOP Factor was only launched in February 2020, we see the low transparency and openness scores in sleep research and chronobiology journals as an opportunity to revisit how we do science, and how we report it. Importantly, while the TOP Factor is a useful instrument to assess transparency and openness at the journal level, it is clear that science policy makers, scientific and learned societies, funders, and research institutions play a key role in incentivizing open science.Another point to discuss would be the potential misuse of the TOP factor, i.e. some form of “gamification” or “open washing”, where the TOP factor becomes the new impact factor (https://go.nature.com/3gat5Vl). It may also be worth discussing whether the TOP factor is enough to encourage good science. Especially in sleep science there is a tendency to collect very small samples and as far as I can see there is no measurement for this included in the TOP factor . Also, funders and universities may play a much more important role in incentivizing open and reproducible science. We have now addressed this in the discussion, where we write:Gaming the TOP Factor?17, 18. As principles of open science are becoming more important in hiring and funding decisions, the TOP Factor may similarly be subject to being ‘gamed’, with ‘openwashing’ being an immediate concern19.  Ultimately, whether or not this is a key driver will only be available for study retrospectively. Quantitative metrics can be subjected to being ‘gamed’, i.e., manipulated in such a way that they no longer correspond to corresponding to the original objective of the metric The authors collected data on the impact factor – it would be interesting to compare the general association between impact factor and TOP factor. However, the sample size  might be too small for a meaningful overview. Then it would be useful to nevertheless offer a scatter plot. We now reported the rank correlation, but there is no convincing evidence. As the data are available in the tables, any interested readers will be able to follow up with their own assessments.The authors rightly point out that journals should adopt hard requirements to encourage scientists to practice open science (page 9). This is then followed by a series of reasons why researchers have not adopted open science practices e.g., it costs time. It might be nice to remind the reader of the positive reasons to adopt open science practices .   We have now revised this and highlighted the positive aspects, and also stated that journal guidelines should be framed positively.It is great that the authors have provided the data. However for the second excel file SuplementaryInformation_S2.xlsx, sheet "comparison", the syntax used does not appear to consistently generate the arrays of the rater ratings – might be worth checking this out to see what is going wrong. It might also be helpful to provide an aggregate sheet in that file as was the case in the SuplementaryInformation_S1.xlsx file.   We provide the second table only as a raw record of the intermediate data, and do not think that this will be useful for readers, but have included it for completeness. We would be happy to remove it.The tables in the manuscript are a bit tedious to read. The authors could consider condensing them a bit. Ideally, the tables would be formatted to be publishable in portrait rather than landscape format. This may be achieved by choosing a smaller font and putting some of the labels into the table legend. In case the publisher intends to reformat these tables, this comment can largely be ignored. Point 7: The formatting of the tables is ultimately up to the publisher and we therefore will leave it to them.


Preprints are gaining visibility in many fields. Thanks to the exponential growth in submissions to bioRxiv, an online server for preprints in biology, versions of manuscripts prior to the completion of journal-organized peer review are poised to become a standard component of the publishing experience in the life sciences. Here, we provide an overview of current challenges facing preprints, both technical and social, and a vision for their future development. Science progresses only at the rate at which we can share information with one another. But as any author of a journal article can attest, formal mechanisms of scholarly communication do not always work efficiently and can be subject to biases –3. Peer The sooner a piece of work can be read, evaluated, and built upon, the faster science moves. And by including a greater diversity of thought in the process of science, higher quality final products emerge. Yet, although our system of publication has superficially transitioned from physical print magazines to online websites, the mechanisms and processes of scientific communication are not much faster or more inclusive than they were in the 19th century.Perhaps the underlying cause for this stasis is the fact that our system of evaluating scientific work—whether for deciding what to read or to whom to award grants and jobs—relies heavily on the reputation of journal titles, and in turn, journal article output is factored into university ranking calculations . ExperimHere, we distill what we’ve learned from our work listening to concerns about, and investigating issues surrounding, preprints. We summarize the current state of support for preprinting in the life sciences, discuss extant needs and challenges, and put forth ideas for future developments.Posting preprints is standard practice in many fields in physics, mathematics, computer science, economics, and other disciplines. Preprints are only now becoming widespread in the life sciences , despiteFirst, in today’s digital world, the idea of composing a manuscript in real-time using collaborative editing tools only to not share it with the community seems increasingly anachronistic.Second, bioRxiv was positioned effectively within the existing publishing paradigm from the start. Founded by veterans of the publishing industry, John Inglis and Richard Sever, bioRxiv quickly established partnerships with a number of journals. These journals not only agreed to consider manuscripts posted as preprints but also established a direct submission pipeline enabling authors to submit to both with one click. Furthermore, perhaps driven by a competitive publishing environment, editors began to invite submission of manuscripts from preprint servers (discussed below). Preprints now represent an opportunity to publishers, in which previous efforts to share science in this way may have been seen as a commercial threat. Direct submission arrangements and anecdotes about manuscript recruitment offered researchers confidence that posting preprints would not endanger their chances of journal publication. Furthermore, the ownership of bioRxiv by Cold Spring Harbor Laboratory, a credible, nonprofit research institute, likely contributed to its resonance with the community of authors and readers.Third, many funders have since provided active support and recognition for preprints. Although the NIH has been involved in preprinting through the Information Exchange Groups of the 1960s and Harold Varmus’s 1999 eBioMed proposal , only reFourth, Twitter created a community that provided visibility to preprints and support to their authors . All of In 1966, a cabal of journal editors “outlawed” Information Exchange Groups (the NIH’s photocopy and mail-based preprint exchange platform), fearing that preprints would damage their business model . Indeed,Perhaps for these reasons, authors continue to use journals even in fields in which preprinting has long been common practice. For example, in physics, 73% of papers on the arXiv can be matched to an article that appears in a journal indexed by Web of Science . AlthougIn fact, preprints are very much complementary to journals, and they offer several tangible benefits for editors and publishers. First, although there is no evidence that the relationship is causal, papers that have been preprinted garner more attention over time ,44,45. PPLOS Genetics has pioneered the formalization of this process with preprint editors [Proc B has adopted the practice as well [Furthermore, preprints offer an efficient marketplace for papers . Althoug editors  and Proc as well . Unfortu as well ,52. In oFinally, preprints relieve pressure on journals. Authors generally would like their papers to be published as soon as possible, leading some journals to promise shorter peer review turnaround times, perhaps at the expense of allowing reviewers to be as thorough as they would like to be . If authJACS, and the American Association for Cancer Research [Journal policies explicitly permitting or even encouraging preprinting have removed much lingering fear of rejection due to prior publication conflicts. Even some long-standing holdouts, notably Cell Press, Research  have updResearch , more inResearch , and detResearch .Preprints allow researchers to demonstrate their most recent work to prospective and current funders. It is becoming less acceptable to cite work that is “in submission” or “under review” in grant applications: when a manuscript is prepared, reviewers wish to see it and may request the applicant cites a preprinted version . PracticIn comparison to journals, university policies for the assessment of applications for hiring, promotion, and tenure seem slower to change , but thePerhaps the most proactive support for preprints has come from funders, who seemed poised to actively encourage the use of preprints in the life sciences. In May of 2016, the Simons Foundation Autism Research Initiative (SFARI) announced it would change its grant award letter to “strongly encourage” investigators to post preprints and that such papers would be taken into consideration in funding decisions . On SeptOne influential funder policy has been NIH’s guide notice NOT-OD-17-050, which clarifies the NIH’s position on preprints and other interim research products: “The NIH encourages investigators to use interim research products, such as preprints, to speed the dissemination and enhance the rigor of their work … Interim research products can be cited anywhere other research products are cited” . A notabSome private funders have gone beyond encouraging preprints to requiring them. Barring privacy concerns, the Chan Zuckerberg Initiative states a commitment to posting preprints prior to peer review . As partAs with all policies, their existence does not ensure they will be enacted. Funders also must develop mechanisms to monitor grantee reaction and compliance. The emergence of technological infrastructure , as well as continued dialogue between researchers and funders, is key to enabling these policies.Some have argued that preprints should be used by funders to achieve open access to the literature in lieu of mandates for open access journal versions , but we At present, preprint servers lack the technological infrastructure that could help them to realize their full potential. Addressing such challenges could make a large impact on how preprints are used and discovered.Biophysical Journal, Plant Direct, and PLOS titles [For example, authors who have previously read a preprint often wish to quickly find out how it has changed upon the posting or publication of a subsequent version. Currently, neither preprint servers nor journals systematically present a summary of the changes made. Some users already make version notes when posting a revised manuscript to bioRxiv; making this more standard practice might involve enabling authors to submit a short piece of text to journals as well, similar to a conflict of interest disclosure or author CRediT declaration. Once this is complete, it would be natural for journals to provide a link back to the preprint version, which would present a more complete picture of how a manuscript evolved over time. Some journals already provide this backwards link—including S titles –70. FormS titles . AlternaF1000 Research and associated platforms such as Wellcome Open Research. Once these papers pass peer review, they appear on PubMed Central along with their date-stamped first version.) This helps to establish a record of what work was done when, irrespective of delays imposed by the peer review process, which is key to determining priority of discovery. Europe PMC already indexes preprints and has implemented links between the preprint and published version of the same piece of work, though improved metadata could facilitate further search and tool development [Change is needed in search tools, too. For example, preprints could be linked from PubMed and PubMed Central. . Adding to the confusion, preprint servers differ in their own policies for manuscript deposit; in many disciplines  preprint servers also host postprints. In the life sciences, PubMed Central, complemented by institutional repositories, fulfills this need, and bioRxiv hosts only preprints, not postprints. However, other platforms can host biology postprints, e.g., OSF Preprints.A common fear cited as a barrier to preprinting is “getting scooped.” Researchers may feel this has happened when a competing research group publishes highly related work without crediting  their own preprint. As a consequence, their work receives less attention and recognition, and if the work is still unpublished, this can mean publication in a “lower” journal. Scooping can of course occur in the absence of preprinting when work is discussed at conferences, submitted to journals, or included in funding applications, or simply by coincidence of two groups working in a related area. Fears of scooping may be particularly acute in fields with “a well-defined objective” such as structural biology, although the actual consequences may not be as severe as feared . PreprinIt stands to reason that scooping fears are most acute when the stakes are high and careers are on the line. However, fears about scooping—and the secrecy that accompanies them—cannot be neatly divided by generations because it’s rare for a group of co-authors to be homogenous in years of experience.Fear of scooping impacts not only researchers’ willingness to share preprints at all but also whether they are willing to share auxiliary materials that are normally shared as a condition of journal publication. For example, communities have yet to come to consensus on whether authors should be obligated to share reagents or strains after posting a preprint. In a future world where preprinting is universally regarded as a respected disclosure, ethical standards of disclosure should match those associated with journal articles.One of the major arguments against the use of preprints is concern about exposing the public to unverified information, especially in fields in which public interest is high, such as medicine . For exaAs preprints rise in popularity, it is important that all readers and especially journalists have access to information about what a preprint server is and the screening checks (or lack thereof) that it employs. Many preprint servers  display a disclaimer explaining the nature of preprints and the fact that they have not yet been “certified” by peer review.The reaction of a community via public commenting can also be enormously helpful. Although the majority of feedback on preprints comes via mechanisms that is harder to tie to the preprint itself , publicMoving forward, it would be helpful to provide all readers with information that outlines the level of scrutiny and/or acceptance by experts that an individual research output has received and for journalists and other public commentators to critique accordingly.As the production of scientific outputs continues to accelerate, both as a result of a growing number of researchers and their increasing willingness to share, we will need new ways of dealing with information overload. Although a glut of publications may feel like a 21st century problem, thinkers since Seneca have lamented the overabundance of information, and scholars have progressively developed tools to help organize and filter it .reimaginereview.asapbio.org. We believe that this emerging space will become an essential component of the preprint ecosystem.Currently, readers report finding preprints by searching for keywords . They also report being alerted to interesting work on Twitter. The first strategy is directed by subject area but not interest, and the second by interest but not necessarily subject area. Ultimately, we will need more efficient ways to combine both search criteria in a single stream, in much the way that journal title is presently used (rightly or wrongly) to help parse search results in PubMed. Rxivist is one such tool that marries current interest and subject area , and we Curation of interesting or highly respected preprints can also improve their usefulness in evaluating scientists for jobs and grants. Although journal name (and Impact Factor) are flawed proxies for judging the quality of a work , they saAlthough the rate of preprints appearing on bioRxiv is increasing exponentially, they still comprise less than 3% of the monthly output of PubMed . BecauseWith reduced gatekeeping mechanisms, preprints could be a mechanism for sharing and consuming the latest science irrespective of social hierarchies. We must ensure that preprint infrastructures and social mechanisms develop with issues of diversity, equity, and decolonialization of scholarship in mind ,96. Who The growing adoption of preprints in biology is being largely driven by researchers in North America and Europe: of the top 100 institutional affiliations ranked by number of preprints posted to bioRxiv until December 2018, only 6 are located outside these regions , whereasReflecting on the “scooping” concerns listed above, we should consider how preprints could offer appropriate recognition and support for creators of openly shared work. Indeed, some researchers report only being rewarded with funding and jobs when they are authors of (high-impact) journal articles and not for reuse of their open data sets . TherefoOnce work is shared openly, it is important to address how widely it is seen. Twitter is a major driver of attention on preprints, and social connections between preprint authors and readers raise visibility in the absence of dissemination through journals. Thus the visibility of preprints is strongly influenced by the authors’ existing network “connectedness” and therefore is vulnerable to the same underrepresentation issues we face elsewhere in science. There have been several initiatives to increase the visibility of underrepresented scientists ; foSome have expressed concern at the roughly 35% of preprints that do not go on to be published in a journal, believing that these preprints must be of low quality . AlternaThese low usage rates suggest that preprints alone are not likely to be a solution to publication bias; our current incentive system does not sufficiently reward investments of energy spent writing up contradictory or confirmatory findings in the format of a journal article. Some of this effort, e.g., carefully assembling a methods section, is necessary to reproduce the work and must not be compromised. But other tasks, like putting the work in context with an introduction or interpreting the findings in a discussion, is less useful to specialized readers, who are the likely audience for contradictory or confirmatory findings anyway. In fact, those readers do not need the element of a narrative (often constructed post-facto) that ties together figures in a traditional paper. In these cases, a single figure  would sThere is presently an expectation that all products appearing on preprint servers are more or less complete articles. This helps to promote an image of the preprint server as a destination for high-quality work and helps to facilitate some very positive behaviors, such as the soliticiation of submissions by journal editors. However, this norm reinforces a culture in which research is shared relatively late in the process and also feeds some behaviors that are less desirable, such as counting the number of papers on a CV as a measure of productivity without assessing their contents. Although this practice makes little sense, it is a real concern, as evidenced by the fact that the Medical Research Council worded its preprint policy to discourage researchers from “salami slicing” their preprints into many smaller units for the purpose of gaming the system by gaining a higher publication count . It is nDespite the apparent benefits of micropublications and preprints, both technical and social innovation is required to address open questions. Namely, how can science be shared in varying orders of detail, complexity, and review status over time, from first observation of a result to acceptance of a generalized finding into broader understanding? Which research outputs  are useful to embed in a narrative article? For which of these outputs is subsequent filtration and curation valuable? Ultimately, where it is most useful to invest resources in coordinated peer review, journal production processes, and dissemination of findings to nonspecialist communities? Regardless of when or how preprints fit into this picture, we should strive to ensure that research integrity is rewarded, discovery is accelerated, and the publication process is more inclusive and equitable.