Publishing peer review materials alongside research articles promises to make the peer review process more transparent as well as making it easier to recognise these contributions and give credit to peer reviewers. Traditionally, the peer review reports, editors letters and author responses are only shared between the small number of people in those roles prior to publication, but there is a growing interest in making some or all of these materials available. A small number of journals have been publishing peer review materials for some time, others have begun this practice more recently, and significantly more are now considering how they might begin. This article outlines the outcomes from a recent workshop among journals with experience in publishing peer review materials, in which the specific operation of these workflows, and the challenges, were discussed. Here, we provide a draft as to how to represent these materials in the JATS and Crossref data models to facilitate the coordination and discoverability of peer review materials, and seek feedback on these initial recommendations. Peer review is the practice of subjecting a scholarly article, such as a research paper submitted to a journal, to scrutiny or review by others ('peers') who are experts in the same field. Generally, if the author of the article addresses the concerns raised during peer review to the satisfaction of an editor, the article is accepted for publication. The peer review process produces a trail of documents, which can include: different versions of the article; the reviewer reports (with or without the name of the reviewer); responses by the author to the reports; and various letters (including cover letters from the author and decision letters from the editor). It is also possible for an article to go through two or more rounds of peer review, which increases the number of documents generated.Traditionally the documents generated during the peer review process were only ever seen by the author, the editor and the reviewers, but a small number of publishers now publish some peer review materials alongside articles. Moreover, support for this practice has been slowly gaining momentum, driven by a wish to increase transparency and provide credit for peer reviewers /Europe PMC, and Crossref discussed the practical challenges involved in publishing peer review materials. We focus on what has to happen after a publisher decides to start publishing peer review materials, and discuss how to do this in a way that is sustainable, improves discoverability, and supports machine readability and archiving. We do not discuss the relative merits of the different approaches to peer review that have emerged over the past decade, notably the many different flavours of 'open peer review' , but we As mentioned above, peer review materials can include: the reviewer reports (with or without the reviewer name); responses by the author to the reports; and various letters (including cover letters from the author and decision letters from the editor). Some articles go through two or more rounds of peer review, which increases the number of documents generated. While each document is usually accompanied by a date, other metadata concerning the correspondence can be highly variable. For example, peer review reports and editors decision letters may or may not include names of reviewers or editors, or their ORCID IDs; the individual materials may or may not have DOIs. In subscription journals it is also possible for peer review materials to appear in front of or behind the paywall.Publishers are approaching the publication of peer review materials in a number of ways. The aims of this group were not to prescribe what should be done from an editorial point of view, but to enable what is published to be found by readers and machines alike. Prior to the workshop, data were collected from each publisher attending and we found the following materials are published:○ Peer review reports, anonymized or with report author names○ Author responses/Rebuttals○ Editor decision lettersSome journals also provide appeal and resubmission information .In some cases publishers make peer review materials available as a single PDF with versioned reports linked to specific revisions of the article. Other publishers create separate artifacts, each with unique DOIs, and still others edit and amalgamate various reports into one narrative. The variety can be found in the meeting notes and a table filled out before the meeting See.https://www.crossref.org/services/content-registration/peer-reviews/). As downstream recipients of peer review materials, PMC and Europe PMC presented the perspective of archives that collaborate with journals to ensure that content is being captured in a sustainable and consistent format that fosters long-term preservation and access to the scholarly record. Understanding the goals of, workflows, and limitations on each stakeholder allowed the group to refine the scope of the discussion and its outcomes.After collecting these data, representatives from each of the identified publishers were contacted to attend a workshop in London on July 6, 2018; all but one publisher was able to attend. Further publisher representatives were invited following communication with ASAPbio – these publishers are embarking on this practice. Crossref, Europe PMC and PMC were also represented at this meeting, as downstream recipients of this content or the metadata related to it. Journal representatives expanded on the data previously collected and shared details on how they collect and publish peer review materials, how these artifacts are represented in the ANSI/NISO Z39.96-2015 Journal Article Tag Suite (JATS) document standard (these principles can be applicable to other DTDs), and whether they send relevant metadata to Crossref. Crossref shared details of their schema extension to represent this form of content . We identified three levels of achieving this, with level 1 being the basic and least preferred option, but probably currently the most achievable and pragmatic:1. Peer review materials are attached to the article as a single or numerous PDFs. Whether these materials are pulled together into one document or attached as separate documents, there should be some defined mechanism in the JATS XML tagging that would support the capture of any available metadata and identify these files in a machine-readable and interoperable way for publishers to tag this content appropriately.2. Peer review materials are appended to the article within the full text  as a sub-article component of the XML.3. Peer review materials are full-text XML “articles” or “commentaries” in their own right that link bidirectionally to the main article.Whether the material is provided as a PDF(s) attachment to the main article, or as a full-text XML sub-article or separate article, important metadata can be attached to the item in a machine-readable way, and DOIs can be applied. What types of peer review information are available is dependent on the publisher’s peer review policy, for instance whether reviewers and editors are named, whether the peer review material carries the same license as the main article or takes another form, and what items constitute the peer review materials. Additional metadata fields, such as dates of review and date of review publication and the inclusion of ORCID IDs for reviewers and editors will also be subject to publisher policies and workflows. However, all of this material can be added to the item in a machine-readable way. Even if the actual content is not published in full-text XML format, the metadata can (and should) be.The topic of licensing of the peer review materials was briefly discussed at the workshop but ultimately left out of the remit of this group because the JATS tagging schema would allow for different licensing information to be added for these items or to retain that of the main article, as a publisher chooses.While a few of the publishers had processes in place to prepare the peer review content automatically or within a few minutes, others spend 20–40 minutes per article. In such cases, the tasks that are attributed to this time include the following:Removing boilerplate text from review reports“Stitching together” the material from disparate locations in the submission systemEditorial checks:Reviewing the content for sensitive information, e.g., unpublished data additions and confidentiality leaks, as well ensuring the tone of the report is appropriateRemoving author responses that contain data the author wants to publish in a subsequent paperArbitration processes for conflicting reviewsWhere time is spent—whether in the editorial or production process—depends on the publisher workflow. Regardless of workflow, the overlap in tasks identified provides evidence of the potential value of updating the infrastructure of submission systems to account for and streamline these efforts. Coordination between publishers and submission systems could minimize the time spent “stitching together” peer review materials into a publishable format.In addition to time and workflow hurdles, another major challenge noted by those publishers without their own hosting platforms, was the actual publication process and online hosting of peer review materials. Many publishers identified that some online hosts were not able to manage this new content type. As a result, peer review materials are being captured in supplementary material sections because alternative options are not available. In such cases, it becomes more difficult to capture any relevant associated metadata in a meaningful way for the peer review materials or to make this valuable content easily discoverable.These challenges are common for publishers in that most of the established submission systems and hosting platforms were designed and built many years ago and may be slow to accommodate new requirements. Coordinated communication with these platforms regarding the workflows around publishing peer review materials may result in more satisfactory and generic approaches to accommodating publication of peer review materials.There are internal challenges of cost control issues that also need to be accounted for, and the publication of a single PDF is often more achievable financially based on current systems than producing full-text XML. However, the attachment of machine-readable metadata to that PDF should be within reach, especially if the submission systems and hosting platforms can build these requirements into their products.Recommendations of the NISO/ALPSP Journal Article Versions (JAV) Technical Working Group .To this list, the JATS4R working group on “Article publication and history dates” added pre-print. The JATS4R draft recommendation advises that if the publisher publishes a revision of any of these stages, the subsequent revisions should be labelled with suffixes, as follows: “-r1”, “-r2”, etc. .Review documents may be supplied as:1. sub-articles <sub-article> to the article being reviewed (sub-articles may be full-text XML or XML metadata plus a link to the PDF)2. independent articles <article> <sub-article> or <article> MUST have an article-type attribute with one of the values listed inThe term manuscript does not map to anything we’ve discussed.This is an optional item. Currently there is no corresponding tag in JATS and so would require a request to the JATS Standing Committee.There would be a fixed value list, mapped to the Crossref schema:major-revisionminor-revisionrejectreject-with-resubmitacceptaccept-with-reservationNOTE: There should be no “recommendation” for author-comment type content.It is an optional element and should be contained within <contrib>, which should contain a <name> or <anonymous/>.If <contrib> is used, it MUST contain @contrib-type that maps to following controlled vocabulary:○ For Peer Reviews, use @contrib-type=“reviewer”○ For Decision Letters, use @contrib-type=“editor”○ For Author Reply, use @contrib-type=“author”We intend that the @contrib-type attribute value reflects the contributor’s relationship to the peer review process and not the relationship with the document.The <role> tag is optional and can be used for display terms of what publishers may use for their journal .https://jats4r.org/authors-and-affiliations).Names, affiliations and contributor IDs (such as ORCID), where provided, follow standard JATS tagging .Each review document  SHOULD have license information with a machine-readable license. Review documents supplied as <sub-article> may have their own <license> element or inherit their license information from the parent document as described in the JATS4R Permissions Recommendations  MUST have a pub-date and may have other publication information captured as <event>. Review documents supplied as <sub-article> may have their own <pub-date> element or inherit their <pub-date> from the parent document.There are some elements that MUST NOT appear in review documents:a. <funding-group>b. <app>, <app-group>, <ack>, <glossary>, <back>/<sec>c. <supplementary-material>, <inline-supplementary-material>d. <bio>e. <article-version> Once published, review documents will not be “versioned”. If the reviewer(s) write a review on an updated version of the manuscript, the peer review is a new published object.https://support.crossref.org/hc/en-us/articles/115005255706-Peer-Reviews.As of November 2017, Crossref supports the scholarly discussions entailed in the publication process as well as those after publication (e.g. “post-publication reviews”). In the same fashion as all content registered with Crossref, peer review metadata is available via the open Crossref APIs and Crossref Metadata Search. For full details and example deposit XML, see the Crossref peer review deposit guide:We also propose that publisher web platforms and archives display peer review materials  in a clearly labeled peer review section. This practice will help ensure that not only are the journal processes transparent but that the content itself is easy to find and navigate to, regardless of how a journal chooses to make them available.This proposal is intended to lay the groundwork for the publication and archiving of peer review materials across publishers and publication models, providing flexible options to meet different journal needs and workflows. Moving forward, there is a need for continued collaboration and discussion as peer review models and workflows evolve. As the goals of these peer review efforts are more clearly defined across the publishing and academic communities, certain models may lend themselves more readily to supporting those desired outcomes. Continued efforts to identify the most critical needs of each user group should be explored through ongoing efforts such as ASAPbio and FORCE11. In turn, these needs can inform the technical solutions and recommendations going forward.Further technical discussions should not be placed on hold in the interim, though. As publishing peer review materials practices grow, there is a pressing need for industry-wide solutions now. We would like to see the XML recommendations from this group be converted to a JATS4R recommendation on the publishing of peer review materials. Similarly, it would be of value to the community for Crossref and JATS to coordinate efforts and ensure some level of metadata alignment for peer review materials that would reduce costs to publishers and minimize barriers to implementation.This type of coordination between publishers, archives, and other organizations that support the scholarly communication enterprise is critical to ensuring that the needs of the whole community are being met. Past experience has taught us that making content available is just the first step toward increasing transparency. Doing so in a flexible, consistent, and meaningful way is imperative in making certain that the available material is also discoverable and that long-term preservation of the content can be supported. Implementing the next steps through community-driven recommendations in a sustainable way will be important in increasing transparency and rigor of the scientific record.If you are publishing peer review materials, or are not yet and are considering doing so, please comment.No data are associated with this article. “Publishing peer review materials” reports the outcomes of a 2018 workshop convened with stakeholders engaged in the publication and dissemination of peer review report, to work towards consensus on “how to represent these materials in the JATS and Crossref data models to facilitate the coordination and discoverability of peer review materials”. The article is valuable for understanding the range of publisher workflows in operation for publishing peer review at present. As such, it both makes the case for the need for standardisation of description and proposes a draft JATS XML proposal designed to aid depositing to Crossref. The proposals are pragmatic and reasonable, reflecting the variety of workflows in operation at present and the possibilities for timely transition towards a more standardised system. This first version clearly indicates that it is a “draft” to “seek feedback on these initial recommendations” – but despite its provisional nature, I believe it is ready to be indexed already in this version, as it is a sufficiently complete record of the discussion to this point. The article makes very clear how much work lies ahead, and that this document is just a starting point. Given its purpose, I don’t believe formal peer review is actually necessary here – as a working proposal released to the community for feedback, the views of each member of the community are equally important. I hope my comments will be received as just one more voice in that conversation, and given equal weight with views shared by other channels. I agree with Reviewer 1’s comments regarding useful ways to enhance the level of detail here. To this I would only add that it would be useful to describe why the JATS elements under the sub-heading “Not allowed” are so - why must they not appear in review documents? Other than this, I would only note some further thoughts for how these discussions could develop in the future. None of these issues need necessarily be considered within this present article, but I record them here in hopes of continuing the conversation:Links to review policies: To understand the context of the review it would be helpful to have some lasting link to review policies under which the review was conducted. For example, what criteria were the reviewer asked to consider? Was it single/double-blind or open identities review? If the latter, were reviewer identities revealed to the author (and if so at what stage)? This information could be made available via descriptive metadata fields in this schema, or via persistent links to policy descriptions.From “is” to “ought”: I understand it is not the scope of this document to prescribe practices, but to ensure pragmatic routes to interoperability of metadata description. Hence, I understand that the “topic of licensing of the peer review materials was discussed at the workshop but ultimately left out of the remit of this group because the JATS tagging schema would allow for different licensing information to be added for these items or to retain that of the main article, as a publisher chooses”. Nonetheless, I think as a community some best-practice guidelines would be helpful to ensure the interoperability and re-usability of this content. This point goes more generally for other best-practice issues . In the “Future/next steps” section you note that discussions should continue via other fora – here, I wonder if there is a need for a formal working group hosted by, for example, Force11, to work towards consensus on such topics. .Reviewer roles: We may in future see more stratification of reviewer roles and it would be interesting to have recorded exactly what parts of a manuscript each reviewer had considered . Also, if any computational tools had been applied to check, e.g., statistics. Here it would be useful to have a standard typology of such roles.Other review materials: There are a multitude of venues for “post-publication” review – including journal article comment sections, third-party platforms like publons or pub-peer, or even individual blogs, etc. It would be useful to have formal, persistent links between such materials and the article in question so that readers may more easily place the article in the context of a continuing discussion. I imagine Crossref event data will be useful for this, but I wonder if there would be value in third-party platforms also applying this schema (suitably extended/adapted) to represent these links in a more formal way?  I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. In “Publishing peer review materials,” Beck and coauthors present the outcome of a workshop that assessed the current state and best practices for the technical representation of peer review material and metadata. The report is well-organized and proposes guidance on XML tagging that the group hopes will be adopted by JATS4R. This article is an important contribution to the conversation about increasing transparency in peer review and will facilitate its development as a valued, linked, and preserved class of scholarship. I don’t feel that any of the following comments need to be addressed before the manuscript is indexed in PubMed, but I do believe that the report could be strengthened by attention to these areas:PresentationThis piece would benefit from more illustration of workshop outputs. For example, is there a file you can point (or embed inline) that demonstrates best practices for tagging?In a similar vein, there is great information in Supplementary File 2. Can the process information be used to generate an in-text table?XML proposalNote that I lack practical experience with JATS.Could the tag also be used to identify student or postdoc co-reviewers?Is intended to represent the publication date of the review (presumably the same as the article) or the (in this case probably more useful to readers) date that the review was submitted? Or is that captured in ? Clarification would be helpful.Why is disallowed? I would like to imagine a future in which peer review is recognized as a valuable scholarly activity, something that funders would like to know they are supporting.Text editsPlease define “BMA.”“Past experience has taught us that making content available is just the first step toward increasing transparency.” What does this refer to?“The term manuscript does not map to anything we’ve discussed.” What does this mean?I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.


The Indus Health Network (IHN) is becoming a major player in Pakistan’s healthcare system. With expanding hospitals, rehabilitation centres, blood banks and public health programs, it provides free-of-cost, accessible, high quality healthcare without discrimination to increasing numbers of ordinary people. ICON, the biennial medical conference hosted by IHN, is a natural beneficiary of this success. It is now a leading platform for sharing excellence in healthcare and research with internationally renowned delegates and a multidisciplinary audience. ICON 2020, the fifth of its series, has an opportunity to go one step further.In this month’s edition of the Pakistan Journal of Medical Sciences, we are delighted to share and highlight biomedical research at IHN. Distinctly translational and collaborative, IHN supports research via the Indus Hospital Research Centre (IHRC), from study design to publication through to initiating novel service delivery application. The scientific, clinical and programmatic advantage of sharing knowledge and experience, be it rare case reports to trials with international collaborators, is thus at the heart of this first ever publication. It aims to consolidate ethical scientific enquiry at IHN with emphasis on multidisciplinary and thematic research to bridge gaps in knowledge and practice.It is more than serendipitous that this supplement was organised to coincide with ICON 2020, themed “Building Bridges for Better Healthcare.” From the onset, the Scientific Committee approached the “Herculean” task with a predetermined process. A call for papers was given in March 2019 and kept open until mid-May 2019. In this short time, a tremendous response of 69 articles was received. After the initial administrative shortlist by the Committee, 44 articles were sent for double blind peer review. Each article had two independent reviewers: a national or international subject matter expert and a research professional (biostatistician or epidemiologist) from IHRC. Third reviews were arranged on specific peer reviewer request. Reviews were strictly confidential and conducted on pre-designed scoring proformas for uniformity. They were shared with authors to guide amendments until peer reviewer’s satisfaction. Submissions then underwent internal editorial review for final selections. In all, 20 manuscripts were chosen for publication, blending 11 original research papers, one review article, a case series, two case reports, two letters to the Editor and a section on continuing medical education.Authors from over 14 medical specialties have come together to create this unique issue. Inevitably some high quality research could not be published due to space constraints or inconsistency with our envisioned theme and purpose. Nonetheless, we congratulate all submitting authors for their efforts irrespective of final outcome. We are grateful to all the reviewers for their time and expertise, indebted to administrative staff who worked behind the scenes and appreciative of the ICON Organizing Committee’s support. We also wish to give special thanks to the Pakistan Journal of Medical Sciences for the publication in their highly reputed journal and guidance in the process.Welcome to ICON 2020! We hope to see you at the Marriott Hotel, Karachi, January 17-19, 2020.


The reward infrastructure in science centres on publication, in which journal editors play a key role. Reward distribution hinges on value assessments performed by editors, who draw from plural value systems to judge manuscripts. This conceptual paper examines the numerous biases and other factors that affect editorial decisions. Hybrid and often conflicting value systems contribute to an infrastructure in which editors manage reward through editorial review, commissioned commentaries and reviews and weighing of peer review judgments. Taken together, these systems and processes push the editor into a role resembling censorship. Editors and authors both experience this phenomenon as an unintended side-effect of the reward infrastructure in science. To work towards a more constructive editor-author relationship, we propose a conversation, an exchange between editor and author in which value is collectively assessed (or constructed) as obligatory passage points in the publishing process are traversed. This paper contributes to the discourse on editorial practices by problematising editorial paradigms in a new way and suggesting solutions to entrenched problems. The production and distribution of scientific credit, responsibility, and accountability has garnered growing attention amongst those pursuing critical enquiries into the epistemic and social organisation of science and the consequences for the knowledge produced as a result . We realise that the topics we deal with in our own line of research, including but not limited to research integrity, publication ethics, the sociology of scientific knowledge and science and technology studies, predisposes us to critical interactions with editors. If anything, this may serve to make valuation dynamics more overt. However, we are very aware that we are presenting a pilot analysis, rather than a fully developed ethics or sociology of editorship.ad verbatim testimonies by the editors we interacted with. Communications between editors and authors are private, as indicated by journals’ terms and conditions and are in some cases explicitly included in every email message: “This e-mail is confidential and subject to copyright. Any unauthorised use or disclosure of its contents is prohibited”. Therefore, rather than contacting editors and risking their refusal to use direct quotations, we paraphrased all the interactions we include below and removed all indicators that may reveal the editor’s, journal’s or publisher’s name. Our paraphrasing preserves the original meaning of editors’ statements although the wording has been changed. This is not the case for public statements or quotes drawn from the public domain. In the context of our pilot analysis this is not an immediate problem, since we seek to initiate a structural understanding of how editors assign value and decide accordingly. Our interactions with editors are not generalisable in an empirical sense, but the lessons we draw from them may inform editor-author interaction around the value or worth of scholarly work in the future.We cannot, however, provide Your arguments in favor of open review and anonymous authorship are both interesting and thought-provoking. I agree that there lies value in a system in which people sign their names alongside their opinions. However, there is also value in a system in which people are free to speak openly without fear for personal consequences. In sum, I am not convinced that the bias and power redistribution your suggestion produces are smaller than the ones currently in place. Entirely anonymous authorship, while possibly attractive, is very impractical. I regret that we cannot consider publication of your paper.Here, the editor is essentially admitting that the paper is not being sent for review because the editor does not find the arguments convincing, despite seeing merit in them. The fact that her position might be due in large part to value systems promoting the current system within which she operates is not mentioned. Shutting down debate in this way, editors do a disservice to the scientific community.We accept that there are valid arguments towards avoiding effort by peer reviewers, and we do work towards this by sharing reviews between journals at X, as you already indicate. However, we are unconvinced that an online database containing all reviews would yield more benefit than harm, given the current publishing model.Here, the reason given is again that the editors do not find the arguments convincing – a judgment ordinarily left to peer reviewers. More specifically, the editor inadvertently concedes that the arguments in the paper point towards changing the model of publishing – again challenging the value systems at play.2Editorial review and the ways in which it shapes publishing are most visible in situations where peer review is either absent or less important. Consider, for instance, large multidisciplinary journals which mainly publish original scientific research, yet for various  reasons also publish significant amounts of front matter, or ‘commentary-type’ articles. While it is true their focus is on original scientific research, which is peer-reviewed, these journals tend to review and reject many submissions internally without sending them out for review. Such decisions can be the result of considerations of quality, but they may also occur because the editors disagree with the content despite recognising its relevance and potential impact. A case in point was when one of the current authors had two rejections from a major science journal because editors disagreed with the conclusions of the content. In the first of these, a commentary discussing alternative authorship cultures, the reason for rejection was, paraphrased, as follows:To ensure efficient review and publication, we have instituted a policy stating that only those submissions that have the approval of all editors are forwarded to the next stage. This policy aims to provide authors with a quick response and avoid delays in case you want to submit your manuscript to a different journal.This effectively means that any paper must meet with the unanimous approval of all editors before even being sent out for peer review. This particular manuscript was suggesting a controversial clinical technique for organ preservation – one unlikely to meet with approval from all editors because of its challenging ethical nature. Allowing each editor a veto over controversial content is likely to stifle innovative ideas. Applying such a policy to the more standard original research articles this journal publishes, would stifle innovative thought, and, by default, disqualify potentially paradigm-shifting work. Next to the policy itself, ethical questions arise on how it came into being, what inspired it, and how such a decision ought to be communicated to potential authors.In both these cases, the editors’ unilateral rejection is particularly ironic given that both manuscripts suggested changes in standard editorial practices. Would they have done this with original research, even if it challenged traditional scientific paradigms? Presumably not, given that the big journals prize novelty – sometimes to a fault, itself testimony to a medial and economic value system being at play. This attitude apparently does not extend to anyone daring to suggest that current editorial paradigms should change. And those current paradigms can be quite exacting. An unnamed intensive care journal rejected one of our papers with the following paraphrased response:Editorial review thus actively and overtly politically preselects the contributions that move on to the second round. Such selection does not only take place via the rejection of manuscripts. Even when manuscripts are willingly and enthusiastically received by an editor, the editor may choose to continue to influence their content. Consider, for instance, an original research manuscript submitted by one of us, which solicited a response requiring a specific rewriting before it was to advance to the next stage: “I do like your topic and general approach. I hope that you will consider expanding this paper –some suggestions are below […]”. The suggestions that followed took up over 2000 words excluding a long list of specific literature to be included. Of course, journal editors are, in most cases, experts in their own right. Their expertise may be equally relevant to the manuscript discussed, as that of peer reviewers. Key differences lie in the coverage of their input  and the power that accompanies it, allowing a body of literature to be ‘tweaked’ even before its quality or value can be evaluated through peer review.New England Journal of Medicine has suggested in an interview that “peer review is only a first step to vetting papers that may be interesting and relevant for readers. After a paper passes peer review, it is then given to a team of staff editors who each have a lot of time and space to go through the submission with a fine-toothed comb. So highly qualified editors, not necessarily peer review, act as the journal’s gatekeepers”  are concerned. We are not the first to suggest that editorial control can amount to censorship. For instance, “Biagioli ties the establishment of editorial peer review to the royal license that was required for the legal sale of printed texts”  reviews at many journals are, in part or in full, commission-only. Editors have the liberty to actively seek out ideas and to actively construct reward and value amongst those ideas.We have in mind not so much the commentaries that often accompany research articles (and which thus must be commissioned) but standalone Comment pieces labelled ‘Opinion’, ‘Commentary’, ‘Viewpoint’ or otherwise and that can be found in many journals. For example, in Nature’s instructions for authors, it is stated that “Unsolicited contributions are not accepted. This is a commission-only section intended to be accessible and appealing to the whole global Nature readership, of all disciplines.”These options further expand the epistemic and political reach of editors – as well as imposing aforementioned value systems upon contributors of these texts, while obscuring them from the readers of the same texts. We can discern three routes by which the epistemic and political reach of editors exerts itself. First, through commission-only contributions they are narrowing the ability of researchers themselves to set the agenda of their field. They will have fewer pages per issue available for original contributions in which they may articulate their own ideas  and have to compete with other genres of contributions. Furthermore, if commentary-type articles are the only or preferred way to get an original idea that is not a typical piece of research into a major journal, then making such pieces commission-only in effect shuts out all other authors, obscuring the status or spread of that idea in the community. Second, commissioned pieces are articulations of the editors’ agenda, meaning that a commissioned piece is very unlikely to take a critical stance towards paradigms dominant in that journal. Third and finally, even if a commissioned writer chooses to challenge orthodoxy, forward novel and critical ideas or provide potentially paradigm-altering work, he or she might find it difficult to persist if the editor who commissioned the piece prefers a different view.Commission-only articles act in tandem with the aforementioned semi-censorship of ideas by editors; on the one hand, editors censor ideas that challenge current scientific or publishing paradigms, while on the other, they commission pieces that promote and extol the virtues of the current dominant paradigms . One example occurred when one of us had an article accepted by a major medical journal, and the editors commissioned an editorial from a politician who appropriated the key concept from the article and used a misunderstanding of it to campaign for a change in organ donation systems. The editors did this without informing the author and creator of the concept in question, who could have pointed out to them prior to publication the problems with the editorial. To their credit, the editors apologised when a complaint was made, but the damage was already done.In addition to epistemic control through preselection and commissioning and influence through pre-review, editors have further tools available to construct specific contents of their journals and distributing the value and rewards associated with them. When a paper is submitted to a journal and subsequently sent out for peer review, editors will be on the receiving end of those reviews. In our considerable experience as authors of over 100 papers, rather than merely forwarding them, editors almost always attach judgements to the set of reviews. This provides assistance to authors, who have to navigate difficult value judgments by reviewers. For instance, if reviews contradict one another in terms of content, an editorial comment may indicate which review the editor considers dominant. If reviews fail to raise a specific issue, the editor may choose to include it nonetheless. If reviews contain different valuations, editors help prioritise them. We observed that in some cases editors themselves act as an informal third or fourth reviewer, making points that neither reviewer mentioned, but which nonetheless must be addressed in order for a paper to be accepted.Editors may disregard or expand reviewers’ recommendations, and while editors are obviously not obligated to follow reviewers’ suggestions, some editorial decisions very clearly contradict the recommendation without sufficient– or indeed any –rationale. Despite endless and valid critique, peer review is justified by arguments that it maintains quality  the major, monolithic journals discriminate against theoretical and conceptual articles by making them commission-only, but more importantly by rejecting critical ideas before peer review merely because editors disagree with the content. If they did this with original articles, imagine the controversy: editors reject results of study because they disagree! However, more subtle censorship infrastructures exist through careful nudging of the content of papers through translation of peer review results, additional editorial review steps (pre- or post-peer review), or authors anticipating all of the above and engaging in self-censorship to navigate the conflicting value systems of the contemporary publication landscape.Of course, the degree to which such semi-censorship is performed will vary greatly across disciplines, journals and regions. The stronger the position one value system is in place vis-à-vis the others, the more it will determine the content of value judgements made by editors. Following Franzen , we hypoThis pilot analysis of editors as gatekeepers of reward has several limitations. First, apart from the literature we cite, it is built entirely on our own personal experiences of interacting as authors with editors . However, this experience is not inconsiderable, and we have attempted to avoid having any personal biases skew our analysis. This is a pilot analysis using our own data, and we suggest that larger analyses using a wider pool of data would be beneficial and, accordingly invite others to join the conversation. Second, we have paraphrased communications from editors rather than quoting them verbatim in order to respect both their confidentiality and the rules of the journals concerned. Despite our attempts to paraphrase carefully, this limits the reader’s ability to assess the meanings of those communications themselves.Editorial valuation practices consist of work, the type of work that goes into assessing contributions, articulating value associated with them internally and translating it to a value judgement that also has merit externally. We propose that editorship, in close allegiance with authorship, should pursue value through dialogue. In practice, this should take the shape of a conversation, a practice in which value is constructed in a manuscript through a series of open interactions in which editorial decisions are less unidirectional decisions and more collective obligatory passage points on the route to publication (and reward)  is opening up its archives of editor-author interactions.More research is needed on the various cultures and paradigms that can bias editorial decisions and lead to conscious or accidental censorship. Specifically, we envision three routes towards expanding our empirical and conceptual understanding of editorship and evaluations of value. First, an empirical analysis of the recently opened ASA archives (as mentioned above) would offer clear empirical examples of author-editor conversations. The ASA archives offer the type of data required for a deeper and more constructive understanding of value creation and value evaluation by authors and editors. Second, a larger survey of authors’ experiences of editors acting as gatekeepers and censors, ideally conducted by independent researchers whose experiences are not included in the analysis, would be beneficial. Any such study would avoid the potential for bias in our pilot analysis while also providing more generalisable data. Third, an ethnography of editors focused on the many value judgements that take place in their practices would allow access to the tacit expertise editors build and transfer, how their relationships to publishers influences these, and the articulations they offer themselves when it comes to their gatekeeping endeavours.In addition, editors should be assisted to assess and avoid (where possible) the potential for dominant paradigms to bias their decisions. Since editors perform their duties largely without any formal training (Galipeau et al.


What claim does your employer have on your publications? On your data? On your intellectual properties? If you teach, research, or publish something controversial, just how much freedom do you have to do what you wish? If you want to leave your institution for new employment, what are you allowed to take with you?https://en.wikipedia.org/wiki/Academic_freedom. Really, this is a good idea not just for universities. A diversity of ideas makes for better businesses and other organizations as well.I considered a number of definitions of academic freedom, and I like this succinct one from Wikipedia the best: “Academic freedom is the conviction that the freedom of inquiry by faculty members is essential to the mission of the academy as well as the principles of academia, and that scholars should have freedom to teach or communicate ideas or facts  without being targeted for repression, job loss, or imprisonment.” https://www.insidehighered.com/views/2010/12/21/defining-academic-freedom.Academic freedom exists first, for the good of the people, then to support academic excellence through diversity, and finally to protect the employment, security, and quality of life interests of the scholars. Academic freedom will mean something different altogether, depending on the individual and the venue. For some, it means the right to say what they wish in a classroom, regardless of the context, but there are limitations to this. As an example, teaching the doctrines of phrenology might be acceptable in a class on historical medicine, but not in a contemporary class on neuroanatomy. Penn State professors made a heroic attempt to further cement the principles for academic freedom in modern academia back in 2010. Here are a few of its main points: Academic freedom means that both faculty members and students can engage in intellectual debate without fear of censorship or retaliation. Academic freedom in teaching means that both faculty members and students can make comparisons and contrasts between subjects taught in a course and any field of human knowledge or period of history. Academic freedom means that the political, religious, or philosophical beliefs of politicians, administrators, and members of the public cannot be imposed on students or faculty.https://osc.hul.harvard.edu/policies/fas/ Many scholarly communities have followed suit.What does academic freedom mean in the context of open access publishing, and what are its implications for ownership of data? Here, we are in the midst of debate and controversy. In 2008, Harvard University faculty voted unanimously “to make available his or her scholarly articles and to exercise the copyright in those articles. In legal terms, the permission granted by each faculty member is a nonexclusive, irrevocable, paid‐up, worldwide license to exercise any and all rights under copyright relating to each of his or her scholarly articles, in any medium, and to authorize others to do the same, provided that the articles are not sold for a profit.” https://www.uni-konstanz.de/en/university/news-and-media/current-announcements/news/news-in-detail/open-access-satzung-auf-juristischem-pruefstand/. Not everyone agrees that articles that do not need to be open access by law should nevertheless be published open access regardless of the desires of the author. Authors may have justifiable reasons to prefer publishing in a venue that does not have unrestricted access, such as the inclusion of intellectual property, controversial data, or political perspectives that are sensitive. Also, the universities’ right to publish all faculty articles open access implies some kind and degree of ownership over the article. This is true even if the copyright is assigned to the authors. Should academics, therefore, be forced to publish open access by university mandate? Who really controls the copyright?However, not all agree that universities have a right to such secondary publication of faculty articles. Seventeen academics at the University of Konstanz have objected and taken the open access policy to the German courts. https://www.nature.com/news/us-court-grants-elsevier-millions-in-damages-from-sci-hub-1.22196. It seems unlikely that Elsevier will see any of these funds, despite that fact, the judgment is likely small compared to the loss of restricted access to their many journal articles and other publications over the years. Also, consider that despite this ruling, Sci‐Hub is very popular. Hence, will anything change? Not likely. What this means for the academics at the University of Konstanz is their apparent quest to avoid publication in open access journals is probably doomed in the long run.In a larger context, the move toward universal open access publication of research papers continues relentlessly. On June 21, 2017, a US court granted Elsevier a $15 million‐dollar judgment against Sci‐Hub for copyright infringement. However, Sci‐Hub is based in Russia and the founder, Alexandra Elbakyan, apparently does not have any assets in the US. Another area of contention is the ownership of the data from which scholarly work is derived, analyzed, interpreted, and eventually described in a publication. If a faculty member moves from one university to another institution, who controls the data and its future use? If the data are based on protected health information (PHI), there are strict legal limitations on how that data may be shared. In this event, usually the university owns the data, even if the data are anonymized. There may be exceptions to this general rule that involve specific university policy governing collaborations with scholars from outside institutions within the legal framework of HIPAA. These would need to be set up in advance of beginning projects involving such data. A scholar is responsible to understand what data may and may not be taken to new employment. If a scholar working with non‐PHI data has spent years building a repository with potentially valuable and significant intellectual property, there may be objections if and when the scholar leaves for other employment. In this case, the outcome would be negotiated on a case‐by‐case basis.All of this is to say that academic freedom, however defined, is evolving and new perceptions and alliances are being formed and negotiated. As scholars, we need to be aware who owns our publications, our data, and our intellectual property. These policies are being hammered out at individual universities, nationally, and globally. It is important that we know these things before we consider new employment, where the local policies and enforcement may be an unwelcome surprise.


There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review’s ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.The online version of this article (10.1007/s11192-018-2969-2) contains supplementary material, which is available to authorized users. There is a growing concern about erroneous or fraudulent research that gets published in the scientific literature. Mainly originating in the biomedical sciences, scholars have demonstrated that a large proportion of published articles contains flaws . A request for retraction may be made by publishers, (co-)authors, research organisations, funders, or any other actor in the research process, but the decision to retract remains an editorial one. Retractions occur for a wide variety of reasons, ranging from honest error to severe cases of research fraud  journals were effectively reached. After sending out the questionnaire, several respondents offered to further disseminate the questionnaire among their networks, which we gratefully accepted. Hence the questionnaire was subsequently distributed among the European Association of Science Editors (EASE), the International Society of Managing and Technical Editors (ISMTE) and through the newsletter of the Committee on Publication Ethics (COPE). We stopped collecting responses on April 5th 2018.After the reminders, we eventually obtained a total of 361 useful responses. The final response rate of 6.12% is low, but comparable to, or even higher, than response rates of similar online surveys among journal editors or authors regarding issues related to academic integrity , based on the classification in research areas of the Leiden Ranking  Errors, Fake Review, and Misconduct (grouped). Misconduct (grouped) is a category encompassing all other categories except (honest) error and further includes the general labels referring to misconduct as given by Retraction Watch. A complete classification of the reasons for retraction as presented by Retraction Watch into the categories specified above, can be found in the supplementary material. Note that a single retraction usually occurs due to various reasons, classifying individual retractions in multiple categories. In addition we note that Retraction Watch also uses the ‘error’ category in cases where other reasons for retraction, such as misconduct, cannot be proven (even though strong suggestion to the contrary might exist).The data from For each of the twelve characteristics of peer review we studied, we analysed whether the review procedure had a (significant) association with the chance of a paper being retracted. Significance was tested using a log likelihood ratio-test on the ratios of retracted and non-retracted papers. In addition, subgroup-specific effects for the five research areas were tested using binary logistic regression as interaction test between the peer review procedure and the research discipline. Here we use the Wald-statistic on the logistic model with the interaction term, compared to the model without the interaction term. Last, specific effects for the various retraction reasons were analysed, using an ANOVA test to analyse differences in mean retraction numbers, and multinomial logistic regression to analyse the effect size of various peer review procedures on retraction rates. Analyses were performed using the SPSS 25.0 statistics software.This section presents the results of our analyses, grouped by the various attributes of peer review presented in Table Traditionally, peer review occurs between the submission and publication of a manuscript. However, over the past decades, new peer review procedures have been proposed for different phases of the publication process. Most notably these include: pre-submission review (e.g. through registered reports) Chambers , in whicΛ(3) = 18.899, p < 0.001). Due to the ambivalent nature of retractions as both indicator of undetected errors or of the willingness to repair errors, this could mean that these journals are less prone to take action after publication. However, since this system is used with the explicit intention to prevent tweaking of data or statistics, it seems highly unlikely that this lower retraction rate is due to lax editorial attitudes towards problematic research.Our results suggest that the pre-submission system is indeed related to fewer retracted articles Table : in totadf = 5, p = 0.364), nor reasons for retraction  = 5.409, p = 0.020).The rates for traditional pre-publication review (97.6% vs. 96.3%) and post-publication review (0.8% vs. 0.0%) did not show significant differences. However, the fact that no retractions were reported in journals using post-publication review is interesting. It might suggest that potential issues are dealt with in review and commentaries, rather than using retractions as a mechanism to correct the literature, but the number of publications reviewed in this way is still relatively low. No significant interactions were found with respect to research area  are used to assess manuscripts. Some journals have deliberately decided BMJ Open . Their rΛ(3) = 18.779, p < 0.001), with significantly more retractions for journals using novelty and anticipated impact as assessment criteria. No significant interactions were observed regarding research area , nor reason for retraction  = 1.220, p = 0.301), suggesting that the effect is homogeneous with respect to research discipline and type of problematic research.The results of our analysis Table  suggest p = 0.016). Apparently, they have ‘more to choose from’ than journals that do not use impact as criterion, and/or have a tighter limit on the number of articles they can publish . However, the higher retraction rates suggest that these journals either attract more problematic submissions or are less capable of filtering them.The higher retraction levels among journals aiming to publish highly relevant and novel research, usually journals with high impact factors, has also been established in previous research on retractions Steen ; He 2013lower 2016 JIF than those which do not . The precise relation between impact as a selection criterion, JIF, and retraction rates would have to be analysed in a larger, multivariate analysis, but our findings suggest the impact criterion provokes more retractions and fails to increase the JIF rating.In addition, it could be expected that the strategy to select articles with the highest anticipated impact would be expressed in a higher journal impact factor (JIF). However, the journals that use anticipated impact as a selection criterion, on average, do not have a higher journal impact factor. On the contrary, journals in our sample using impact as a selection criterion have a slightly The use of external reviewers, i.e. researchers not directly affiliated with the journal, did not become standard practice until well after WWII Baldwin . Still tΛ(5) = 116.527, p < 0.0001), with relatively few retractions occurring when editors-in-chief or the wider community are involved in review  = 10.538, p < 0.001): when the editor-in-chief, the editorial committee or author-suggested reviewers are involved, relatively few retractions appear for fake review reports or issues with references, while relatively more retractions occur for authorship or ethical issues. Finding relatively few retractions for fake peer review when author-suggested reviewers are used, is somewhat puzzling, as these types of actors seem most vulnerable to fraud with review reports. More research will be needed to elucidate the mechanism underlying this association.Our analysis shows a significant impact of the actor type performing the review  always aware of authors’ identities, whereas authors knew the identity of the editor-in-chief, but not necessarily of the editorial committee or invited outside reviewers (single-blind review). Responding to issues of equality and fairness (Zuckerman and Merton Λ(2) = 106.042, p < 0.0001). The effect can be witnessed in all research areas, but is especially strong in the social sciences and humanities. In this research area, 79% of all articles went through double-blind review, whereas only 13% of all retracted articles went through this review procedure. In contrast, only 19% of the articles were reviewed in a procedure allowing reviewers to see authors’ identities, whereas 87% of all retractions went through such review. The figures for biomedical and health sciences show a similar, but weaker relation . For the other research areas similar trends were found, but no significant differences occurred. In addition, significant differences occurred when comparing the various reasons for retraction  = 10.630, p = 0.001), with the strongest effects for the category of retractions due to fake review, ethical violations, and misconduct.We analysed the impact of blinding author identities to editors and/or reviewers Table . The resStudies in psychology and economics have previously suggested that people are more strict when reviewing or judging the unknown rather than the known or the familiar  = 5.964, p = 0.0494). Neither do we find significant differences when correcting for research fields  nor reasons for retraction  = 2.839, p = 0.784). This might mainly be due to the fact that an overwhelming majority of the articles, as well as the retractions, goes through the same review procedure: a system in which reviewer identities are blinded to all relevant actors. Hence, to properly study the influence of this review attribute other research strategies such as randomised trails or other intervention studies could be employed.Our data Table  does notIn addition to disclosing reviewer identities, open review frameworks have proposed to also make the review reports accessible. We distinguish four levels of accessibility: review reports accessible (1) to authors and editors, (2) to other reviewers of the same manuscript, (3) to readers of the published manuscript, and (4) to the wider public, i.e. without restrictions  = 9.081, p = 0.0128). However, we did find some specific influences when regarding research area  and the reason for retraction  = 6.897, p < 0.001). Making review reports accessible not only to authors and editors, but also to other reviewers of the same manuscript was associated with fewer retraction due to fake reviews and issues with references, while in this case we see an increase in the rate of retractions due to plagiarism, falsification, image and/or data issues and ethical violations. The fact that no significant effects were measured for the other two review procedures, those in which reports are shared with the manuscript’s readers or the wider public, might again be due to the low number of articles and retractions going through these review procedures. Again, other research set-ups could be employed to study the effect of making review reports more or less widely accessible on the quality of review.In our data Table  we foundBesides sharing review reports or disclosing identities, some journals have introduced review procedures in which interaction between various actors in the review process is facilitated. This includes modest levels of interaction by allowing reviewers to read author responses to their review report, but also goes further by facilitating interaction between reviewers of the same manuscript  = 126.4, p < 0.0001). More specifically, allowing no interaction reduces the likelihood of retractions for fake review, ethical issues and misconduct in general. Contrarily, allowing authors to respond to review reports increases the likelihood of retractions occurring for fake review, ethical concerns or issues with references  = 21.269, p < 0.001). Research area was also found to be a significantly mediating factor  with stronger effects in the biomedical and health sciences as well as the physical sciences and engineering.The data from our study Table  actuallyIn particular, it might be deemed surprising that interaction between reviewers is not associated with lower retraction rates, as more interaction is expected to lead to higher scrutiny during review and hence to fewer retractions. Indeed, in other settings, such as detecting medication errors, it has been suggested that higher levels of cooperation and interaction would be beneficial for effective error detection  questions or to rate specific aspects of the manuscript; semi-structured, when reviewers are presented a list of guiding questions or criteria that might assist them in writing their review; and unstructured, when reviewers receive a manuscript for review without further guidance about review criteria.Λ(2) = 58.907, p < 0.0001), with fewer retractions appearing in either structured and unstructured review, but more retractions appearing after semi-structured review. Specifically, semi-structured review is related to significantly more retractions for fake review, authorship and ethical issues and concerns over references  = 12.538, p < 0.001). In addition, subject area turned out to be a significant mediating factor , with particularly strong effects in Social Science and Humanities and Mathematics and Computer Science, and relatively weak effects in Life and Earth Sciences.Our data suggests Table  that theInterestingly, both extremes of the spectrum appear related to fewest retractions. This suggests that either guiding reviewers very specifically through the review process or leaving them to decide on appropriate ways of reviewing themselves is most effective in detecting problematic publications. Alternatively, partly guiding reviewers seems to be least effective. We could speculate that reviewers in this case would only consider those aspects referred to in their checklist, while editors might expect them to take more aspects of the manuscript into account. However, other mechanisms might also be at play. To obtain a better understanding of this phenomenon, future research could compare specific guidelines for reviewers with the retraction rates on a more qualitative level.Since especially highly structured review procedures were introduced expressly with the intent to address problematic research, it seems improbable that lower retraction rates are to be seen as an indication of unwillingness to address problematic research. However, low retraction rates for the other side of the spectrum, i.e. unstructured review criteria, are harder to interpret in this way.Statistical analyses are increasingly recognised as a source of error, questionable research practices, or outright fraud in quantitative scientific papers Altman . Hence, Λ(4) = 138.858, p < 0.0001). However, the results do not provide evidence for the effectiveness of assigning specialist statics reviewers or employing digital tools to assist in statistical review. Specifically, we witness more retractions appearing in journals that state that statistics is not relevant for their journal, while less retractions appear in journals either paying ‘no special attention’ to review, incorporate review in the standard tasks of reviewers, or use specialist statistical reviewers. A significant difference between research areas was witnessed , suggesting stronger effects in the physical sciences and engineering as well as the life and earth sciences.Our data indeed show Table  a signifF = 63.503, p < 0.001). On the contrary, we do not see any substantial influence on the number of retractions due to errors or issues related to data, which arguably are more related to statistics. The effect of specialist, incorporated or IT-assisted statistics review on aspects of the manuscript directly related to data analysis remains open for further study.When focussing on the different reasons for retraction, the data show that incorporated statistical review is associated with a significantly lower number of retractions due to fake review, authorship- and ethical issues  journals redirect a rejected manuscript to another  journal, along with the review reports, allowing the new journal to quickly decide on the manuscript’s quality, without having to perform another round of reviews Barroga . Other pΛ(3) = 42.270, p < 0.0001). No differences were observed between research areas , nor between reasons for retraction  = 4.166, p = 0.016), hence suggesting similar effects in all research areas and for all types of problematic research.Our data Table , suggestThe fact that no significant differences were found for review reports from commercial platforms or the wider community might be attributed to the low number of articles going through these kinds of review. Hence the effect of those review procedures remains to be studied. The positive effect of sharing review reports with partner journals on the number of retractions is promising, in the sense that sharing review reports potentially not only lowers the burden on the review system, but also improves the quality of the published literature. Here too, since external sources are typically used by journals trying to improve peer review, lower retraction rates are unlikely to be a sign of low willingness to act against problematic research, but rather of a high capability to detect it.One of the most promising innovations in peer review’s error and fraud detection is probably the introduction of digital tools such as plagiarism detection software, image manipulation software, software to check references (for instance for references to retracted articles), or software to assist in statistical review. Such digital tools have been implemented in a wide variety of journals with specific detection objectives  = 42.270, p < 0.0001). In particular, more retractions occur when articles were reviewed without the assistance of digital tools and when (only) software to scan references was used. In this, subject area was a significant mediating factor , with stronger effects in the Social Sciences and Humanities. In addition, the usage of various digital tools has a specific effect on different reasons for retraction  = 27.990, p < 0.001). When no tools are used, we witness more retractions for plagiarism and falsification, while those retractions are sparse when plagiarism detection software is used. Similar to previous attributes, these lower retraction rates seem unlikely to be due to lax editorial attitudes towards problematic research.Our data Table  indeed sIn contrast, when software to check references is used, we witness more retractions for fake review and for issues with references. The latter is clearly contrary to what should be expected, but might be explained by the sensitivity of these journals to issues with references, making them more willing to file retractions for such reasons. Here, higher retraction rates might hence be a sign of a more pro-active policy in using retractions to address issues with problematic research.Another way of testing the effectiveness of digital tools is by comparing submissions to journals prior to and after the installation of digital tools. Because the number of changes in review procedures is relatively small, we can only meaningfully perform such an analysis specifically for plagiarism scanning tools. For this case, our results show that journals installing plagiarism software published 70,097 articles prior to the introduction of the software, leading to 38 retractions, 11 of them for plagiarism or duplication. These same journals published 41,043 articles after the introduction of the software, leading to 19 retractions, of which only 1 for plagiarism or duplication. Even though these numbers are still relatively small, it does suggest that the introduction of plagiarism software is an effective way of preventing retractions, specifically for reasons of plagiarism or duplication.PubPeer  = 108.759, p < 0.0001). This suggests that higher scrutiny by readers does indeed increase the detection likelihood of problematic research reports that slipped through review, thereby leading to more retractions. In addition, we find significant differences between research fields  and various reasons for retraction  = 26.607, p < 0.001). In particular we find strong effects in the Biomedical and Health Sciences as well as in the Life and Earth Sciences. Regarding reasons for retraction, peer review procedures with in-channel reader commentary is associated with fewer retractions due to fake review and issues with references, while there are more retractions for falsification and image/data issues compared to review procedures without direct reader commentary.Our analyses Table  demonstrThe fact that more retractions appear when readers are able to comment on articles, suggests that reader commentary is a way to flag issues and put the retraction mechanism in motion. This might hence be a specific effect of how journals deal with errors in the literature. Whereas issues might be addressed internally, or in closed communication with the authors, this becomes more difficult when errors have been publicly announced and reported in reader comments. Again, higher retraction rates might hence be a sign of heightened willingness to address issues with problematic research by means of retractions. However, the extent to which reader commentary leads to retractions should be researched in more detail.Combining the results from the previous sections, Table Our research project may have suffered from various limitations. First, some selection and response bias may have been introduced in journal review data collection. When sampling editors’ email addresses, we searched the Web of Science for editorials written in 2017. Hence only journals indexed in the Web of Science were included. This may have caused some young, non-English, or smaller, niche journals to be excluded from our sample. In terms of publications numbers or retractions, this arguably has little effect on our general results. However, some of these excluded journals may have particularly innovative review procedures, potentially underestimating the spread of peer review innovations.On the other hand, from the journals sampled our survey, those paying specific attention to their review process, and those particularly keen on innovating their review procedures, were arguably most likely to respond to our survey. Hence, from the journals indexed by Web of Science, we expect to have obtained responses exactly from the most innovative journals with respect to peer review procedures. Indeed, our data suggests this, with substantially more innovations reported in journals that responded very quickly to our survey request, compared to journals responding later, after one or two reminders.To assess the effectiveness of different peer review procedures to detect fraudulent or erroneous research we used retracted journals articles as a proxy of problematic publications. We did not consider the number of published errata or corrections because these are not collected as systematically and reliably as the number of retracted articles. This approach has some limitations. First, it allows us only to trace problematic research articles that have been identified as such. Doubtless, many articles containing errors have not been retracted, either because the errors have not yet been identified, because editors are hesitant about retraction measures, authors are successfully fighting retraction measures, because of ambivalences or disagreement about what constitutes error, or a host of other reasons. Hence, our control group of non-retracted articles contains  papers that should have been among the group of problematic articles. Nevertheless, we expect this number to be relatively small compared to the size of the control group, hence only moderately affecting our results.not pass through peer review, i.e. those manuscripts in which errors were identified during the review process. The validity of our findings hence rests on the assumption that problematic articles were submitted via a uniform distribution to journals using different peer review models, i.e. that erroneous research was not submitted with higher probability to journals holding a specific review procedure.Second, by using retracted journal articles, we employ problematic articles that slipped through peer review. With this method we were not able to identify problematic manuscripts that did A final limitation of our study rests in the survey approach to collect data about peer review procedures. Even though we tested our survey before distributing it, this type of data collection is prone to misunderstanding of the wording or confusion about definitions, as well as incomplete knowledge of the editors of (past) review procedures. This might have led editors to classify their journal’s review procedures differently from how we intended it and hence have influenced our analyses. Procedures reported by editors may not always reflect review practice.With these limitations, we believe our research nevertheless provides valuable recommendations for journal editors to effectively design or reconsider their review procedures.In spite of various calls for more research on the effectiveness of various peer review procedures, actual evidence is rare. This study addresses precisely this knowledge gap. Our analyses reveal major differences between various ways of organising peer review and the number of retracted journal articles going through these review procedures. Thereby, they provide an indication of the effectiveness of various peer review models in detecting erroneous or fraudulent research. Even though hard causal connections cannot be made, our data suggest that some review procedures are significantly more effective in preventing retractions. In particular, author blinding seems more effective than reviewer blinding; involving the wider community in review seems beneficial; using digital tools to assist review is related to fewer retractions, as is constraining interaction between authors and reviewers, and using pre-submission review procedures such as registered reports. In addition, our data suggest differences in the effectiveness of various review procedures between scientific disciplines, as well as between specific reasons for retraction. Thereby we present a systematic comparison of review procedures’ effectiveness in detecting problematic research publications.Our results provide specific recommendations and guidance for journal editors and publishers on how to improve the ability of their review processes to detect forms of problematic research that are particularly relevant in their subject area (with the understanding that preferences for particular peer review procedures are informed by many other considerations besides the prevention of retractions). For example, image manipulation or issues related to authorship may be of particular concern to journals in specific research areas. Our findings provide suggestions for editors to organise peer review in order to address exactly those issues, thereby allowing them to tweak review models to their specific demands.In addition, our results suggest directions for future research in order to identify and assess the specific mechanisms underlying the effects of different review procedures. Future research should elucidate the causal connections behind the identified strong correlations. What specific mechanism, for instance, makes double-blind review better capable of detecting errors in research records than single-blind review? And what causes augmented interaction between authors and reviewers to be related to more retracted journal articles? A closer look at the practice of peer review, and particularly in those cases that led to retractions, could clarify this further.  To allow for such future initiatives, we call on journals to be more transparent about their editorial policies and review procedures. Providing such information on journals’ webpages would allow for more inclusive analyses, strengthen the power of the analyses and thereby lead to more detailed results.Much uncertainty exists about which peer review innovations actually work. Our assessment of peer review procedures addresses this knowledge gap and may provide valuable assistance to journal editors, publishing houses, and even funding agencies. We hope this will also contribute to improved research integrity.Supplementary material 1 (PDF 98 kb)Supplementary material 2 (DOCX 14 kb)Below is the link to the electronic supplementary material.


The Times, They are A‐Changin…Several years ago, the Federation of American Societies For Experimental Biology (FASEB) decided to found a new journal to further serve the broad spectrum of members of its constituent societies and the varied aspects of biological sciences they represent. FASEB BioAdvances was officially launched in January of this year under the leadership of Jasna Markovac, the Chair of the FASEB Publication and Communications Committee, and is now approaching the mid‐point of its first year of publication. It is a fully open access journal and is, of course, open to submissions from anyone, with or without a connection to the Federation. In addition to reporting new research findings, FASEB BioAdvances is interested in enhancing scientific discussion and will consider papers that address reproducibility (or the lack thereof) or provide a forum for reporting negative results and the subsequent discussion thereof. It will also publish reviews of timely subjects presented in a balanced way or perspectives/opinion pieces that make a case for a new idea or support an old one.In a time when there is a clear proliferation of publishing opportunities, it is reasonable to ask “Why add to this growing mélange? Aren't there enough journals out there?” We believe the answers to those questions are that science is growing, the scientific community is expanding  and a significant number of these new publishing opportunities can be classed as “predatory”, that is, published for profit without regard for the accepted standards of the scientific community. The Federation, in its wisdom, believes that there is a need for the type of journal that FASEB BioAdvances represents.With FASEB BioAdvances successfully established, Jasna is now passing the baton to me, and I will take over as Editor‐in‐Chief on June 1. Happily she will remain on board as Founding Editor and will continue to supply sound advice as the journal moves forward. This transition will also mark a period of expansion with new additions to the ranks of both the associate editors and the editorial board. In so doing we, the editorial team, hope to expand the scope of the science covered and make FASEB BioAdvances an even more attractive place for publishing original work and thoughtful opinion pieces. We in turn promise to provide timely and meaningful reviews and responsive management of the editorial process. FASEB passed its century mark just a few years ago and it represents one of the most important alliances in the biological sciences. Our challenge is to make this journal equal in stature to the organization that sponsors it.


The theme of this year’s Peer Review Week, Quality in Peer Review, reflects both the necessity of peer review and the growing uncertainty about its role in scholarly publishing. We support peer review that aims to improve manuscripts through critical evaluation before publication. Peer review has been the subject of much discussion recently, particularly given that many researchers now choose to disseminate their work via online preprint servers (such as bioRxiv) prior to, or even instead of, traditional publication in a journal. Some have asked whether it’s necessary to have a manuscript evaluated by 2–3 experts before it can even see the light of day. Wouldn’t it be better to get it out quickly, and let the scientific community judge it for themselves?There is merit to this idea. After all, traditional peer review is often slow and the quality of the reviews is unpredictable. There is no arguing the fact that quite a few papers are published with glaring errors and major flaws that were not caught by the reviewers or editors. But in our experience as editors, many more errors are identified during the prepublication peer review process and this leads to more robust papers making their way to journal pages. Of course, the evaluation of research by one’s peers doesn’t end at publication. Our position is that the less formal post-publication evaluation process cannot substitute for rigorous prepublication peer review.Communications Biology, we try to draw attention to the importance of quality peer review by highlighting some of our most constructive, thorough, and helpful referees on our Reviewer of the Month page.When done well, peer review moves science forward. Independent journal editors help avoid potential conflicts of interest between reviewers and author. Unbiased reviewers provide useful critical feedback that authors can use to improve their work before sharing it with a broader audience. The purpose of peer review shouldn’t be to weed out all errors—this is close to impossible. Rather, it should help authors see their manuscript through the eyes of other experts while giving editors a clearer view as to whether the paper meets their journal’s criteria. At When done well, peer review moves science forward.written before about the need for reviews to be fair and unbiased. Our Referees page also includes detailed guidelines for reviewers and links to reviewer training modules. But at the end of the day, it’s up to the reviewers themselves to approach the task with an open mind and a desire to help their colleagues produce the best possible science. It’s also up to the editors to request clarifications, provide clear guidance to authors, and (in rare cases) to redact improper comments. Editors can also help limit excessive cycles of review and revision.But how do we ensure the quality of peer review? We have Communications Chemistry and Communications Physics. Follow the hashtag #CommsChatPRW to read the comments and add your own. Finally, as always, we encourage you to comment on this editorial below to tell us your views on quality peer review.The bottom line is that peer review works—but there is always room for improvement. This week, in honor of Peer Review Week, we are asking you to tell us: what does quality in peer review mean to you? We have published blog posts on various Nature Research community sites (communities.nature.com) featuring conversations with our Editorial Board members, who are authors, reviewers, and readers, in addition to their role as editors. To gain feedback from a wider community, we hosted a live Twitter chat this Thursday, September 19th, together with our sister journals


Royal Society Open Science and the Royal Society of Chemistry. The collection spans the range of the chemical and material sciences, reflecting the breadth of research areas receiving Royal Society grant support.We introduce 14 articles published as part of the ‘New talent’ special collection of invited articles to showcase some of the exciting work being funded by the Royal Society. As Royal Society University Research Fellows or Dorothy Hodgkin Fellowship holders, the contributors to this collection are rising stars in their areas of research. This collection also illustrates the close collaboration between  It was the first to operate open peer review at the Society and is our first truly cross-disciplinary journal. It is also the first to be published in collaboration with another learned academic Society, the Royal Society of Chemistry (RSC). This collection of articles has its genesis in that collaboration and in the support the Royal Society provides to new talent through its University Research and Dorothy Hodgkin Fellowships.To the best of our knowledge, the collaboration is unique in scientific scholarly publishing. Under many circumstances, two academic Societies, each with interests in publishing chemistry research content, would not be collaborators but competitors. The goal of the collaboration is to provide a joint outlet for the two Societies to allow them to publish high-quality chemistry and material science research, regardless of its subjective impact.Royal Society Open Science has been managed by the RSC.1 A subset of the journal's Associate Editors with expertise in chemistry, led by Anthony Stace as Subject Editor, work with the RSC to conduct the peer review of papers submitted to Royal Society Open Science in chemistry and allied fields such as material science. Any paper eventually accepted is published on the journal's website, and includes an RSC badge to acknowledge the collaboration.Since 2015, the peer review of all chemistry content submitted to 2 Dorothy Hodgkin Fellowships, meanwhile, are awarded to equally promising early career researchers who require additional flexibility in their working patterns, owing to their personal circumstances.3The publications in this collection each include a holder of a Royal Society University Research Fellowship or a Dorothy Hodgkin Fellowship. These awards are granted via a competitive grant application process by the Royal Society. University Research Fellowships are awarded to ‘outstanding scientists who are in the early stages of their research career and have the potential to become leaders in their field’.Royal Society Open Science was a strong endorsement of the high quality of the research these individuals are conducting. Furthermore, the curation of this ‘New talent’ special collection provided an opportunity for the journal to work closely with the wider Royal Society to establish a ‘proof of principle’ for future ‘New talent’ collections in other major areas of research. Lastly, and to reiterate, the collection provides a vindication of the opportunities presented when two learned Societies work on a joint venture providing a template for new collaborations in future.Inviting these future leaders to submit to http://royalsocietypublishing.org/cc/new-talent.4 The contributions span across the range of theoretical and experimental chemical and material sciences, both original research and reviews, reflecting the breadth of research supported by the Royal Society. As readers will see, the collection represents a wide swathe of chemical research from inorganic to metalorganic to biological and analytical chemistry.The collection comprises 14 papers, curated at Royal Society Open Science.We have enjoyed reading and preparing these manuscripts for publication, and we hope that readers of the journal will also find something of interest in these papers. As Editors of the journal, we are grateful for the support provided by these Fellows, and we hope this is the beginning—rather than the conclusion—of continuing relationships between them and


Verbatim, the full review was “It is incredible if it is true.” I cherish it as a souvenir and use it as a perfect example of what must not be done. Quality in peer review is this year’s topic for Peer Review Week.After having spent months/years doing experiments and analyses, we are finally ready to tell our story to our fellow scientists. However, before knowledge is transmitted to others, we must pass under the yoke of the review process. In many instances, it is painful, stressful, and even, sometimes, humiliating. The “best” review I ever received was from one of the two What is a good-quality review? The answer is surprisingly easy: a good review is a helpful and useful one. When early career scientists come to see me when they have a paper to review, sometimes for the first time, I tell them that the only thing they have to do is to check whether the interpretations/conclusions are supported by the presented data and analyses. If there are some issues, they must try to help the authors provide a better case without asking them to do unnecessary experiments.eNeuro is based on these basic principles. The reviewing editors pay great attention to what is transmitted to authors: reviews must be factual, not emotional, and should include improvement suggestions (if necessary). If the reviewers agree that more experiments are needed and that experiments will require more than two months’ work, the paper is automatically rejected (with the possibility to resubmit). This procedure allows researchers to really ponder which additional experiments are truly necessary. Finally, reviewers and the reviewing editor must reach a consensus on what comments will be transmitted to authors. Therefore, the authors receive a one-voice factual report. This provides a clear directive toward the path to publication and eliminates the need for authors to try to interpret the priorities of separate reviewers. Sometimes, generating one consensus review requires several exchanges and discussion between the reviewers and the reviewing editor, dialogue is the key to success. We know that the system works as, since the launch of eNeuro in 2014, I can count on two hands the number of appeals I have received. Even if one may be unhappy after rejection, the decision is accepted because the facts and reasons are provided. I am 100% convinced that this type of reviewing (pioneered by eLife) is today’s best solution to the concerns raised by traditional peer review. It is easy to implement, but it takes more time per manuscript. For obvious reasons, it works best if the reviewing editor is an active scientist. We also know that eNeuro’s system works based on the positive comments we receive from authors (included those with rejected papers) and reviewers regarding the quality of our peer review process. Before hopefully becoming the norm, mentalities must change.Our review process at https://www.jneurosci.org/content/sfn-reviewer-mentor-program) to train graduate students, postdocs, or established researchers to write good and helpful reviews; trainees are then invited to become reviewers at eNeuro. When you are a reviewing editor, you may think that you are taking a risk when selecting a non-seasoned reviewer. But so far, reviewing editors who have used trainees from the program have been enthusiastic regarding the quality of their reviews. I believe that this will induce a virtuous circle. The happier authors become with the review process, the better their own reviews will be, making even more authors happy, etc. I am not overly optimistic; it is working at eNeuro. We have the opportunity to shape the future of the publication field. Let us seize it.The best way forward is to teach the young generations the fundamentals of a good-quality review. Unfortunately, there are few teaching courses provided by research institutions on how to review a paper. The Society for Neuroscience offers a mentorship program (And for those who have read the Guide, you know that the final answer is to be found on page 42.