Background: “Open peer review” (OPR), despite being a major pillar of Open Science, has neither a standardized definition nor an agreed schema of its features and implementations. The literature reflects this, with numerous overlapping and contradictory definitions. While for some the term refers to peer review where the identities of both author and reviewer are disclosed to each other, for others it signifies systems where reviewer reports are published alongside articles. For others it signifies both of these conditions, and for yet others it describes systems where not only “invited experts” are able to comment. For still others, it includes a variety of combinations of these and other novel methods.Methods: Recognising the absence of a consensus view on what open peer review is, this article undertakes a systematic review of definitions of “open peer review” or “open review”, to create a corpus of 122 definitions. These definitions are systematically analysed to build a coherent typology of the various innovations in peer review signified by the term, and hence provide the precise technical definition currently lacking.Results: This quantifiable data yields rich information on the range and extent of differing definitions over time and by broad subject area. Quantifying definitions in this way allows us to accurately portray exactly how ambiguously the phrase “open peer review” has been used thus far, for the literature offers 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature reviewed.Conclusions: I propose a pragmatic definition of open peer review as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process. They don’t have precise or technical definitions. No matter how they’re defined, there’s a large area of overlap between them. If there’s ever a difference, some kinds of open review accept evaluative comments from any readers, even anonymous readers, while other kinds try to limit evaluative comments to those from ”peers“ with expertise or credentials in the relevant field. But neither kind of review has a special name, and I think each could fairly be called “open review” or “open peer review”.” - Peter Suber, email correspondence, 2007As with other areas of “open science” , OPR hasIn short, things have not improved much since Suber made his astute observation. This continuing imprecision grows more problematic over time, however. As Mark Ware notes, “it is not always clear in debates over the merits of OPR exactly what is being referred to” . DifferiTo resolve this, I undertake a systematic review of the definitions of “open peer review” or “open review”, to create a corpus of more than 120 definitions. These definitions have been systematically analysed to build a coherent typology of the many different innovations in peer review signified by the term, and hence provide the precise technical definition that is currently lacking. This quantifiable data yields rich information on the range and extent of differing definitions over time and by broad subject area. Based on this work, I propose a pragmatic definition of OPR as an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process.Peer review is the formal quality assurance mechanism whereby scholarly manuscripts  are made subject to the scrutiny of others, whose feedback and judgements are then used to improve works and make final decisions regarding selection . Peer review usually performs two distinct functions: (1) technical evaluation of the validity or soundness of a work in its methodology, analysis and argumentation (answering the question “is it good scholarship?”), and (2) assisting editorial selection by assessing the novelty or expected impact of a work . The two processes need not be entwined, and some journals such as PLOS ONE and PeerJ, have begun to adopt models where reviewers are asked to focus only on technical soundness.per se is necessary, but most find the current model sub-optimal. Ware’s 2008 survey, for example, found that an overwhelming majority (85%) agreed that “peer review greatly helps scientific communication” and that even more (around 90%) said their own last published paper had been improved by peer review. Yet almost two thirds (64%) declared that they were satisfied with the current system of peer review, and less than a third (32%) believed that this system was the best possible , have the potential to address such issues.in 2008 , a figurin 2008 . These cLack of accountability and risks of subversion: The “black-box” nature of traditional peer review gives reviewers, editors and even authors a lot of power to potentially subvert the process. At the editorial level, lack of transparency means that editors can unilaterally reject submissions or shape review outcomes by selecting reviewers based on their known preference for or aversion to certain theories and methods . Open peer review would also improve communication. Referees and authors could discuss difficult issues to find ways to improve a paper, rather than dismissing it. Frequently, the review itself provides useful information. Should not these contributions be shared? Interested readers should have access to the reviews of the published papers.” “[O]pen review makes submissions OA [open access], before or after some prepublication review, and invites community comments. Some open-review journals will use those comments to decide whether to accept the article for formal publication, and others will already have accepted the article and use the community comments to complement or carry forward the quality evaluation started by the journal. ” Within just these two examples, there are already a multitude of factors at play, including the removal of anonymity, the publishing of review reports, interaction between participants, crowdsourcing of reviews, and making manuscripts public pre-review, amongst others. But each of these are distinct factors, presenting separate strategies for openness and targeting differing problems. For example, disclosure of identities aims usually at increasing accountability and minimizing bias, c.f. “referees should be more highly motivated to do a competent and fair review if they may have to defend their views to the authors and if they will be identified with the published papers” . Publicadefining factor (open identity), which leaves us where we started. Ford’s schema is also problematic elsewhere: it lists “editor-mediated review” and “pre-publication review” as distinguishing characteristics, despite these being common traits of traditional peer review; it includes questionable elements such as the purely “theoretical” “synchronous review”; and some of its characteristics do not seem to be “base elements”, but complexes of other traits – for example, the definition of “transparent review” incorporates other characteristics such as open identities (which Ford terms “signed review”) and open reports (“disclosed review”).This diversity has led many authors to acknowledge the essential ambiguity of the term “open peer review” . The majTo resolve this ambiguity, I performed a review of the literature for articles discussing “open review” or “open peer review”, extracting a corpus of 122 definitions of OPR. I first searched Web of Science (WoS) for “TOPIC: (”open review" OR “open peer review”)”, with no limitation on date of publication, yielding a total of 137 results (searched on 12th July 2016). These records were then each individually examined for relevance and a total of 57 were excluded. 21 results  had been through an OPR process (which was mentioned in the abstract) but did not themselves touch on the subject of OPR; 12 results used the phrase “open review” to refer to a literature review with a flexible methodology; 12 results were for the review of objects classed “out of scope” ; 7 results were not in the English language; and 5 results were duplicate entries in WoS. This left a total of 80 relevant articles which mentioned either “open peer review” or “open review”.th July 2016) were examined to find references in “grey literature”  and books respectively. Finally, the author examined the reference sections of identified publications, especially bibliographies and literature reviews, to find further references. Duplicate results were discarded and the above exclusion criteria applied to add a further 42 definitions to the corpus. The dataset is available online (http://doi.org/10.5281/zenodo.438024).The same search terms were applied to find sources in other academic databases . In addition, the first 10 pages of search results for these terms in Google and Google Books  was given, implicit definitions were gathered from contextual statements. For instance, “reviewers can notify the editors if they want to opt-out of the open review system and stay anonymous”  is takenIn searching only for the terms “open review” and “open peer review”, the study explicitly limits itself only to that literature which uses these terms. It is hence important to note that it is likely that other studies have described or proposed innovations to peer review which have aims similar to those identified by this study. However, if they have not explicitly used the label “open review” or “open peer review” in conjunction with these systems, those studies would necessarily fall outside of scope. For example, “post-publication peer review” (PPPR) is clearly a concept closely-related to OPR, but unless sources explicitly equate the two, sources discussing PPPR are not included in this review. It is acknowledged that this focus on the distinct usages of the term OPR, rather than on all sources which touch on the various aims and ideas which underlie such systems, limits the scope of this study.The number of definitions of OPR over time show a clear upward trend, with the most definitions in a single year coming in 2015. The distribution shows that except for some outlying definitions in the early 1980s, the phrase “open peer review” did not really enter academic discussion until the early 1990s. At that time, the phrase seems to have been used largely to refer to non-blinded review (i.e. open identities). There is then a big upswing from the early-mid 2000s onwards, which perhaps correlates with the rise of the rise of the openness agenda  over that period . Most ofSixty-eight percent (n=83) of the 122 definitions identified were explicitly stated, 37.7% (n=46) implicitly stated, and 5.7% (n=7) contained both explicit and implicit information.The extracted definitions were examined and classified against an iteratively constructed taxonomy of OPR traits.Open identities: Authors and reviewers are aware of each other’s identityOpen reports: Review reports are published alongside the relevant article.Open participation: The wider community are able to contribute to the review process.Open interaction: Direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.Open pre-review manuscripts: Manuscripts are made immediately available  in advance of any formal peer review procedures.Open final-version commenting: Review or commenting on final “version of record” publications.Open platforms (“decoupled review”): Review is facilitated by a different organizational entity than the venue of publication.The core traits are easily identified, with just three covering more than 99% of all definitions: Open identities combined with open reports cover 116 (95.1%) of all records. Adding open participations leads to a coverage of 121 (99.2%) records overall. As seen inIf we break down these traits by the disciplinary-focus of the definition source, we observe some interesting differences between STEM- and SSH-focused sources . Of thosThe various ways these traits are configured within definitions can be seen inThe distribution of traits shows two very popular configurations and a variety of rarer ones, with the most popular configuration (open identities) accounting for one third  and the second-most popular configuration  accounting for almost a quarter  of all definitions. There then follows a “long-tail” of less-frequently found configurations, with more than half of all configurations being unique to a single definition.I next offer a detailed analysis of each of these traits, detailing the issues they aim to resolve and the evidence to support their effectiveness.Open identity peer review, also known as signed peer review  and “unbProponents of open identity peer review argue that it will enhance accountability, further enable credit for peer reviewers, and simply make the system fairer: “most importantly, it seems unjust that authors should be “judged” by reviewers hiding behind anonymity” . Opponenhttp://embor.embopress.org), review names are published alongside the reports. The main benefits of this measure lie in making currently invisible but potentially useful scholarly information available for re-use. There is increased transparency and accountability that comes with being able to examine normally behind-the-scenes discussions and processes of improvement and assessment, and a potential to further incentivize peer reviewers by making their peer review work a more visible part of their scholarly activities .Open reports peer review is where review reports (either full reports or summaries) are published alongside the relevant article. Often, although not in all cases , or some credentials might first be required . Open participation is often used as a complement to a parallel process of solicited peer review. It aims to resolve possible conflicts associated with editorial selection of reviewers  and possibly improve the reliability of peer review by increasing the number of reviewers . This haCriticisms of open participation routinely focus on questions about reviewers’ qualifications to comment and the incentives for doing so. Given that disciplines are subject to increasingly narrow specialization, especially in the sciences , it can Nature, for example, ran an experiment from June to December 2006 inviting submitting authors to take part in an experiment where open participation would be used as a complement to a parallel process of solicited peer reviews. Nature judged the trial to have been unsuccessful due to the small number of authors wishing to take part (just 5% of submitting authors), the small number of overall comments  and the insubstantial nature of most of the comments that were received (Atmospheric Chemistry and Physics (ACP), which publishes pre-review discussion papers for community comments, only about one in ﬁve papers is commented upon .Another issue for open participation is that difficulties have been reported in motivating self-selecting commentators to take part and deliver useful critique.eceived . At the Open interaction peer review allows and encourages direct reciprocal discussion between reviewers, and/or between author(s) and reviewers. In traditional peer review, reviewers and authors correspond only with editors. Reviewers have no contact with other reviewers, and authors usually have no opportunity to directly question or respond to reviewers’ comments. Allowing interaction amongst reviewers or between authors and reviewers, or between reviewers themselves, is another way to “open up” the review process, enabling editors and reviewers to work with authors to improve their manuscript. The motivation for doing so, according to , is to “EMBO Journal, for example, enables “cross-peer review,” where referees are “invited to comment on each other’s reports, before the editor makes a decision, ensuring a balanced review process”  allow authors to short-cut the traditional publication process and make their manuscripts immediately available to everyone. This can be used as a complement to a more traditional publication process, with comments invited on preprints and then incorporated into redrafting as the manuscript goes through traditional peer review with a journal. Alternatively, services which overlay peer-review functionalities on repositories can produce functional publication platforms at reduced cost  in advance, or in synchrony with, any formal peer review procedures. Subject-specific “preprint servers” liken arXiv . The recn arXiv . AnotherElectronic Transactions in Artificial Intelligence (ETAI), where a two-stage review process was used. First, manuscripts were made available online for interactive community discussion, before later being subject to standard anonymous peer review. The journal stopped publishing in 2002  before the start of the peer review process. This approach was pioneered with the 1997 launch of the online journalin 2002 .Atmosph review . Other pThe benefits to be gained from open pre-review manuscripts is that researchers can assert their priority in reporting findings – they needn’t wait for the sometimes seemingly endless peer review and publishing process, during which they might fear being scooped. Moreover, getting research out earlier increases its visibility, enables open participation in peer review , and perhaps even, according to , increasOpen final-version commenting is review or commenting on final “version of record” publications. If the purpose of peer review is to assist in the selection and improvement of manuscripts for publication, then it seems illogical to suggest that peer review can continue once the final version-of-record is made public. Nonetheless, in a literal sense, even the declared fixed version-of-record continues to undergo a process of improvement  and selection .Mendeley,ResearchGate andAcademia.edu, viaTwitter, or on their own blogs. In this sense, peer review can be decoupled not only from the journal, but also from any particular platform. The reputation of a piece of work is continuously evolving as long as it remains the subject of discussion. Thus, considering final-version commenting to be an active part of an ongoing, perpetual process peer review in a wider sense hence might encourage an adjustment in our conception of the nature of peer review, away from seeing it as a distinct process that leads to publication, and Improvements based on feedback happen most obviously in the case of so-called ‘living’ publications, like theLiving Reviews group of three disciplinary journals in the fields of relativity, solar physics and computational astrophysics, publishing invited review articles which allow authors to regularly update their articles to incorporate the latest developments in the field. Here, even where the published version is anticipated to be the final version, it remains open to future retraction or correction. Such changes are often fueled by social media, as in the 2010 case of #arseniclife, where social media critique over flaws in the methodology of a paper claiming to show a bacterium capable of growing on arsenic resulted in refutations being published in Science. TheRetraction Watch blog is dedicated to publicizing such cases.The internet has hugely expanded the range of effective action available for readers to offer their feedback on scholarly works. Where before only formal routes like the letters to the journal or commentary articles offered readers a voice, now a multitude of channels exist. Journals are increasingly offering their own commentary sections.Pubpeer which proclaims itself a “post-publication peer review platform”. When its users swarmed to critique aNature paper on STAP (Stimulus-Triggered Acquisition of Pluripotency) cells, PubPeer argued that its “post-publication peer review easily outperformed even the most careful reviewing in the best journal. The papers’ comment threads on PubPeer have attracted some 40000 viewers. It’s hardly surprising they caught issues that three overworked referees and a couple of editors did not. Science is now able to self-correct instantly. Post-publication peer review is here to stay” .Open platforms peer review is review facilitated by a different organizational entity than the venue of publication. Recent years have seen the emergence of a group of dedicated platforms which aim to augment the traditional publishing ecosystem by de-coupling review functionalities from journals. Services likePublons seeks to address the problem of incentive in peer review by turning peer review into measurable research outputs. Publons collects information about peer review from reviewers and publishers to produce reviewer profiles which detail verified peer review contributions that researchers can add to their CVs. Overlay journals likeDiscrete Mathematics, discussed above, are another example of open platforms. Peter Suber . However, in opening up participation to the wider community, it might actually increase engagement by those with conflicts of interest. Where anonymity is possible, this may be particularly problematic. Moreover, lack of anonymity for reviewers inopen identities review might subvert the process by discouraging reviewers from making strong criticisms, especially against higher-status colleagues.Social and publication biases:Open reports adds another layer of quality assurance, allowing the wider community to scrutinize reviews to examine decision-making processes. However,open identities removes anonymity conditions for reviewers (single-blind) or authors and reviewers (double-blind) which are traditionally in place to counteract social biases .Lack of incentives:Open reports linked toopen identities enable higher visibility for peer review activities, allowing review work to be cited in other publications and in career development activities linked to promotion and tenure.Open participation could in principle increase incentives to peer review by enabling reviewers to themselves select the works that they consider themselves qualified to judge; however in practice, experience to date suggests that reviewers are less likely to review under this condition.Wastefulness:Open reports make currently invisible but potentially useful scholarly information available for re-use, as well as providing young researchers a guide  to help them as they begin to do peer review themselves.This synthesis allows us to draw the following conclusions: (1) the individual traits of OPR can be argued to address many of the problems with traditional peer review, but (2) differing traits addresses differing problems in differing ways, (3) no trait addresses all problems, and in fact (4) individual traits may actually exacerbate problems in some areas. Assessing this already complex landscape is made yet more problematic by the fact that (5) there is often little evidence to support or challenge many of these claims. There is hence a pressing need for more research to empirically evaluate the efficacy of differing traits in resolving these issues.The traits that we have identified to be part of definitions of OPR are disparate in their aims and implementation. Is there any common thread between them? I would argue yes: they each aim to bring peer review more into line with the emergent agenda of Open Science. To advance this argument, I’ll next briefly describe this movement and its underlying aims, and then relate each OPR trait to this agenda.Open Science is the name given to a broad movement to reshape scholarly communication. As the English word “science” traditionally excludes the humanities and social sciences, the phenomenon is often referred to by more explicitly inclusive terms like “open scholarship” or “open research”. As “Open Science” is the more common term, I shall use it here, but should be read as referring to research from all academic disciplines.Open Science encompasses a variety of practices, usually including areas like open access to publications, open research data, open source software/tools, open workflows, citizen science, open educational resources, and alternative methods for research evaluation including open peer review . These aDemocratic school: Believing that there is an unequal distribution of access to knowledge, this area is concerned with making scholarly knowledge (including publications and data) available freely for all.Pragmatic school: Following the principle that the creation of knowledge is made more efficient through collaboration and strengthened through critique, this area seeks to harness network effects by connecting scholars and making scholarly methods transparent.Infrastructure school: This thread is motivated by the assumption that efficient research requires readily available platforms, tools and services for dissemination and collaboration.Public school: Based on the recognition that true societal impact requires societal engagement in research and readily understandable communication of scientific results, this area seeks to bring the public to collaborate in research through citizen science, and make scholarship more readily understandable through lay summaries, blogging and other less formal communicative methods.Measurement school: Motivated by the acknowledgement that traditional metrics for measuring scientific impact have proven problematic , this strand seeks “alternative metrics” which can make use of the new possibilities of digitally networked tools to track and measure the impact of scholarship through formerly invisible activities.The traits of OPR, in differing yet overlapping ways, each aim to bring greater transparency, accountability, inclusivity and/or efficiency to the restricted model of traditional peer review. The traits of OPR can be fit into Fecher & Friesike’s Open Science schema thus:Democratic school:Open reports further make scholarly products available to all.Pragmatic school:Open identities foster increased accountability by linking scholars’ names to their judgements;open reports increases transparency by opening review reports to readers;open interaction fosters increased collaboration between authors, reviewers and editors in the process of evaluation and revision of scholarship;open pre-review manuscripts enable the earlier dissemination of results.Infrastructure school:Open platforms can make peer review more efficient by decoupling it from journals.Public school:Open participation andfinal-version commenting bring greater inclusivity to peer review by expanding the potential pool of reviewers, including to those outside traditional research actors.Measurement school:Open identities,open reports andopen platforms  enable peer review activities to be more clearly monitored and taken into account in impact-measurement activities.We have seen that the definition of “open peer review” is contested ground. My aim here has been to provide some clarity as to what is being referred to when this term is used. This is especially important since interest in the term (measured via references in the literature) is growing rapidly. By analyzing 122 separate definitions from the literature I have identified seven different traits of OPR, which all aim to resolve differing problems with traditional peer review. Amongst the corpus of definitions there are 22 unique configurations of these traits, meaning 22 distinct definitions of OPR in the reviewed literature. Across all definitions, the core elements are open identities and open reports, with one or both elements present in over 95% of the definitions examined. Among the other elements, open participation is the next most common element, and should perhaps be considered a core trait in SSH. Further secondary elements are open interaction and pre-review manuscripts. Fringe elements include open final version commenting and open platforms.Given that OPR is such a contested concept, in my view the only sensible way forward is to acknowledge the ambiguity of this term, accepting that it is used as an umbrella concept for a diverse array of peer review innovations. Although it could be argued that merely accepting the status quo in this way does not help resolve possible confusion regarding usage, I would argue that quantifying the ambiguity of usage and mapping the distinct traits enables future discussion to start from a firmer basis that (1) acknowledges that people often mean different things when they use this term, and (2) clarifies in advance exactly which OPR traits are under discussion.By being clear about these distinct traits, it will enable us to treat the ambiguity of OPR as a feature and not a bug. The large number of possible configurations of options presents a tool-kit for differing communities to construct open peer review systems that reflect their own needs, preferences and goals. The finding that there seems to be a difference in interpretations between disciplines  reinforces this view. Moreover, disambiguating these traits will enable more focused analysis of the extent to which these traits are actually effective in countering the problems they are claimed to address. This is particularly urgent because, as we have seen, there is often little evidence to support or refute many of these claims.Based upon this analysis I offer the following definition:OPR definition: Open peer review is an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the aims of Open Science, including making reviewer and author identities open, publishing review reports and enabling greater participation in the peer review process. The full list of traits is:Open identities: Authors and reviewers are aware of each other’s identityOpen reports: Review reports are published alongside the relevant article.Open participation: The wider community to able to contribute to the review process.Open interaction: Direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged.Open pre-review manuscripts: Manuscripts are made immediately available  in advance of any formal peer review procedures.Open final-version commenting: Review or commenting on final “version of record” publications.Open platforms (“decoupled review”): Review is facilitated by a different organizational entity than the venue of publication.The data referenced by this article are under copyright with the following copyright statement: Copyright: © 2017 Ross-Hellauer Thttp://doi.org/10.5281/zenodo.438024 . Its provenance is uncertain, even to Suber himself, who recently advised in personal correspondence (19th August 2016): “I might have said it in an email (as noted). But I can’t confirm that, since all my emails from before 2009 are on an old computer in a different city. It sounds like something I could have said in 2007. If you want to use it and attribute it to me, please feel free to note my own uncertainty!” I read the new version and confirm all my comments to the original version are now addressed properly in the article. I have no further comments and approve the current version of the article.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. I commend the author for a thorough revision of the manuscript in the light of reviewers' comments.  I had three significant concerns previously. The first was about methodology, and the author has addressed this with more details and description of the method. The second concerned omitting mention of 'real time peer review in the open' as an option, and including 'open platforms'. Realtime open review is now mentioned. Open platforms are retained, but it is mentioned that they are an unusual (or 'fringe') case of the meaning of open peer review. The final major concern was about conflating editorial selection with peer review, and the author has now nicely clarified this distinction.   My more minor concerns have also all been addressed.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. I commend the author on the edits and revisions to this article. They have strengthened the work. The expansion of the methods section, as well as the new figure and additional language discussing differences between STEM and SSH disciplines enhances this article and situates it for broader, cross-disciplinary discussion and impact. Further, these revisions have allowed the author to articulate areas for further study.  These revisions have addressed the majority of my previous comments. Those comments that were not address by the revisions pertaining to disciplinary economic disparities introduced by article processing charges as well as further delving into differences between STEM and SSH disciplines, are out of scope for this article. Related to the article's discussion of the "wastefulness" of traditional peer review models, my hope is that this thread, which introduces and discusses areas for future research along this vein, is captured in the scholarly conversation and written record.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The definition of open science needs to be clearly stated in the Introduction in order to strengthen the frame of the whole paper. Is the definition you are using of open science fully accepted and not contested? If so, then great, but if not, then it becomes murkier and you might want to spend time unpacking the tension there. Also in the last sentence of the Intro, what is that ethos of open science? Introduction: BackgroundWould it be useful to unpack some counter arguments on the reasons peer review in its current state of blinded does not work? For example, in the delay and expense portion, how does flipping the model to use APCs change the cost at all? And what happens to unfunded research when the model is flipped? Does that create a disparity that only well funded research is readily available? This might create yet another stratification of scholarly publishing and science communication, which one would assume open science is trying to diminish. I realize that you do some of this in the discussion section, but I find there is a gap in the discussion of the economic argument. Introduction: Contested MeaningI appreciate your thoughtful criticisms of past works that have been unable to do what you are doing in this article. Being the author of one of them, however, I would like to make some points.Would like to point out that while I understand the lack of a definition in my authored article (Defining & Characterizing Open Peer Review - 2013) is problematic, it was never my intent to fully describe it, but I had to use a scope for my systematic review, and that scope was identity disclosure. Please note that in my concluding remarks on that paper that I recommended a definition be more tightly defined, and it never claimed to define it wholesale.Your research does a good job picking up the task that other papers were unable to accomplish. MethodologyI would like to hear more in your methodology section about the searching for and selection of social sciences and humanities literature, as I think there might be some gaps in your data set based on this approach. You provide your search terms for Web of Science, but not the other databases and search engines. Including this would strengthen your methodology section. To me, the treatment of social sciences and humanities in this study is one of its weaknesses.Please outline the limitations of your research method.The methods section should be strengthened for better understanding of social sciences and humanities approaches, as well as limitations, for the paper to be more scientifically sound.IntroductionI am not a statistician, nor am I a quantitative researcher, so I cannot provide a robust review of your results when it comes to these facets.Figures and tables are helpful to translate findings and ideas presented. ResultsThis section is well organized and easy to understand. I appreciate the presentation of criticisms of OPR in this section.Social Epistemology, that may help. It would be good to engage more deeply with the question: Is OPR changing the role and purpose of peer review itself? There seems to be evidence of this by the mentorship offered at eLife, the encouragement of reviewers to engage with one another at Frontiers, and generally by collaborative approaches to review that OPR enables. In my view these approaches make peer review more robust, including more than just vetting, fact checking, and some substantive critical feedback. To this end, you will need to more clearly define in your introduction and throughout the paper the assumed purpose of peer review, which you offer us in the open final-versioning commenting portion of the discussion section.Open participation and open interaction sections would be greatly enhanced and do a great service to your consideration of the social sciences and humanities disciplines if you engaged with Fitzpatrick’s work presented in the Mellon White paper as well the Logos article. Additionally, there is an article not included in your data set (was it out of scope?), inOpen platforms section: While I agree that today platforms are an enormous part of our work in communicating science and engaging with our colleagues across the globe. That being said, I would like to point out that the process of peer review could be completely decoupled from a platform. The reason I mention this is that for some individuals and perhaps some disciplines, it might be difficult to get one’s head around the distinction between a peer review process and its technological implementation. To me they are distinct, and it is merely digital technology that assists us in allowing OPR to unfold. It behooves shy away from techno determinism when it comes to the possibilities presented by OPR. ConclusionI think you have a solid finding, but I would like to point out one more quibble. “Open science” is not a term embraced in the social sciences and humanities. Again, since you are couching your definition under the ethos of open science you will need to better describe open science, and make a bridge for social science and humanities disciplines. If an overarching definition of OPR is to be fully accepted by all disciplines, it needs to be inclusive of all of them. This is wherein the tension lies, where the community-based aspect of OPR in the social sciences and humanities  are much more pronounced in the meaning making of the process. How can you better acknowledge the disciplinary tensions in the paper? Or would you like to scope your findings differently? Final thoughtsThis paper is well written and organized logically, which make it quite readable and easy to follow. The main weakness of your paper is the lack of nuance addressed between STEM and social sciences and humanities disciplines. Engaging in the the tension between the approaches to and understanding of peer review and OPR in different disciplines will greatly strengthen your paper.Presenting better the limitations of your method and clarifying your method as noted above will help scope the paper to be more scientifically sound.Finally, clearly define and scope Open Science so that your proposed definition is more understandable. This will greatly strengthen not only the paper, but the definition itself. Discussion:I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Emily Ford: “Introduction: The definition of open science needs to be clearly stated in the Introduction in order to strengthen the frame of the whole paper. Is the definition you are using of open science fully accepted and not contested? If so, then great, but if not, then it becomes murkier and you might want to spend time unpacking the tension there. Also in the last sentence of the Intro, what is that ethos of open science?”Tony Ross-Hellauer: I'd like to thank the reviewer for their very thoughtful and helpful comments. The inclusion of more consideration of the SSH perspective, especially, definitely strengthens the paper.EF: “Introduction: Background - Would it be useful to unpack some counter arguments on the reasons peer review in its current state of blinded does not work? For example, in the delay and expense portion, how does flipping the model to use APCs change the cost at all? And what happens to unfunded research when the model is flipped? Does that create a disparity that only well funded research is readily available? This might create yet another stratification of scholarly publishing and science communication, which one would assume open science is trying to diminish. I realize that you do some of this in the discussion section, but I find there is a gap in the discussion of the economic argument.”TRH: Open Participation relies to an extent on OA (have added a sentence on this), but I’m afraid I don't see a further connection here. Although OPR is of course related to OA journals (in that they have tended to be more likely to experiment with OPR), surely if the same system of  peer review is in use, the basic costs (for review) will be the same? I agree that a fully-APC based OA model of publishing has the potential to exclude less well-resourced institutions , but do not follow how this wider argument is connected to OPR. In any case, I believe these considerations fall out of scope of this review .EF: "Introduction: Contested Meaning - I appreciate your thoughtful criticisms of past works that have been unable to do what you are doing in this article. Being the author of one of them, however, I would like to make some points. - would like to point out that while I understand the lack of a definition in my authored article (Defining & Characterizing Open Peer Review - 2013) is problematic, it was never my intent to fully describe it, but I had to use a scope for my systematic review, and that scope was identity disclosure. Please note that in my concluding remarks on that paper that I recommended a definition be more tightly defined, and it never claimed to define it wholesale. - Your research does a good job picking up the task that other papers were unable to accomplish."TRH: Thanks for clarifying this.EF “Methodology - I would like to hear more in your methodology section about the searching for and selection of social sciences and humanities literature, as I think there might be some gaps in your data set based on this approach. You provide your search terms for Web of Science, but not the other databases and search engines. Including this would strengthen your methodology section. To me, the treatment of social sciences and humanities in this study is one of its weaknesses. Please outline the limitations of your research method. The methods section should be strengthened for better understanding of social sciences and humanities approaches, as well as limitations, for the paper to be more scientifically sound.”TRH: I have expanded the methodology section to better specify search terms and databases used, and included a statement regarding limitations of the search strategy.EF: “Results - I am not a statistician, nor am I a quantitative researcher, so I cannot provide a robust review of your results when it comes to these facets. Figures and tables are helpful to translate findings and ideas presented.”TRH: No response required.EF: “Open participation and open interaction sections would be greatly enhanced and do a great service to your consideration of the social sciences and humanities disciplines if you engaged with Fitzpatrick’s work presented in the Mellon White paper as well the Logos article. Additionally, there is an article not included in your data set (was it out of scope?), in Social Epistemology (http://dx.doi.org/10.1080/02691728.2010.498929), that may help. It would be good to engage more deeply with the question: Is OPR changing the role and purpose of peer review itself? There seems to be evidence of this by the mentorship offered at eLife, the encouragement of reviewers to engage with one another at Frontiers, and generally by collaborative approaches to review that OPR enables. In my view these approaches make peer review more robust, including more than just vetting, fact checking, and some substantive critical feedback. To this end, you will need to more clearly define in your introduction and throughout the paper the assumed purpose of peer review, which you offer us in the open final-versioning commenting portion of the discussion section.”TRH: I have added consideration of disciplinary differences in the results and discussion sections. In particular, I have added a new figure to show the breakdown of traits by discipline, and added more consideration of the philosophical reasons to consider open participation and interaction. The question of whether OPR is changing the role of peer review per se is an excellent one, but I feel it is out of scope for this paper . The article by Fitzpatrick in Social Epistemology was deemed out of scope as it does not mention OPR (save for one mention in a block of quoted text) - the ideas underlying "peer-to-peer review" are no doubt related to the idea of OPR, but the scope here is only those papers which discuss OPR and give an explicit or implicit definition.EF: “Open platforms section: While I agree that today platforms are an enormous part of our work in communicating science and engaging with our colleagues across the globe. That being said, I would like to point out that the process of peer review could be completely decoupled from a platform. The reason I mention this is that for some individuals and perhaps some disciplines, it might be difficult to get one’s head around the distinction between a peer review process and its technological implementation. To me they are distinct, and it is merely digital technology that assists us in allowing OPR to unfold. It behooves shy away from techno determinism when it comes to the possibilities presented by OPR.”TRH: This is an important point - I have added a sentence to the section on final version commenting "In this sense, peer review can be decoupled not only from the journal, but also from any particular platform."EF: “Conclusion - I think you have a solid finding, but I would like to point out one more quibble. “Open science” is not a term embraced in the social sciences and humanities. Again, since you are couching your definition under the ethos of open science you will need to better describe open science, and make a bridge for social science and humanities disciplines. If an overarching definition of OPR is to be fully accepted by all disciplines, it needs to be inclusive of all of them. This is wherein the tension lies, where the community-based aspect of OPR in the social sciences and humanities  are much more pronounced in the meaning making of the process. How can you better acknowledge the disciplinary tensions in the paper? Or would you like to scope your findings differently?“TRH: I have added an explicit reference to Open Science, specifically noting that I use the term to include all academic disciplines. I've also added reference to the disciplinary differences found, and included a concluding note that this area needs further research.EF: “Final thoughts - This paper is well written and organized logically, which make it quite readable and easy to follow. The main weakness of your paper is the lack of nuance addressed between STEM and social sciences and humanities disciplines. Engaging in the the tension between the approaches to and understanding of peer review and OPR in different disciplines will greatly strengthen your paper. Presenting better the limitations of your method and clarifying your method as noted above will help scope the paper to be more scientifically sound. Finally, clearly define and scope Open Science so that your proposed definition is more understandable. This will greatly strengthen not only the paper, but the definition itself.”TRH: Restatement of the above points - see answers above. Tony provides an overview of different definitions of Open Peer Review, acknowledging the ambiguity of the term “open peer review” and the probable impact of such ambiguity on evaluation of the efficiency of open peer review. The author has created then seven OPR traits based on WoS data driven taxonomy. In conclusion though he suggests accepting the existing ambiguity is the only way: “given this is such a contested concept, in my view the only sensible way forward is to acknowledge the ambiguity of this term, accepting that it is used as an umbrella concept for a diverse array of peer review innovations”. This doesn’t seem to be solving the issue he raises at in beginning. On the contrary considering all peer review innovations as forms of OPR might worsen the current situation. Initiatives like “registered reports”, “shortening review deadlines” or “reviewer recognition”, which are meant to address result-biased peer review, incentivizing reviewers, and boosting review speed, respectively, are certainly peer review innovations but cannot be directly considered as part of the umbrella term of open peer review. Hence the article deserves a stronger conclusion, including perhaps a suggested guideline of clarifying what type of “open peer review” beforehand of any evaluation/discussion using authors seven classifications. Author can surely strengthen the conclusion by highlighting the lack of evidence in efficiency claims about some of the 7 traits mentioned in the article. I suggest a double checking the term “Post-publication peer review” in WoS search results. Author reports he has used “Open Final version commenting” for this process in making taxonomy reporting only 6 results. However, Post-publication peer review seem to produce a higher number of documents when I searched the term in Scopus results in 55 documents (11 for 2015 and 19 for 2016). This might impact the result reported in Fig. 6 reporting unique configurations of OPR. Although the author reports 22 different definitions there, the figures shows 23 of them. It is also worth mentioning most of OPR initiatives mentioned in the article are not directly addressing all of shortcomings of the current peer review process. Mainly because one should distinguish between editorial process and peer review process. Issues such as delay and expense are universal to single/double/open peer review as they are part of the editorial process.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Bahar Mehmani: „Tony provides an overview of different definitions of Open Peer Review, acknowledging the ambiguity of the term “open peer review” and the probable impact of such ambiguity on evaluation of the efficiency of open peer review. The author has created then seven OPR traits based on WoS data driven taxonomy. In conclusion though he suggests accepting the existing ambiguity is the only way: “given this is such a contested concept, in my view the only sensible way forward is to acknowledge the ambiguity of this term, accepting that it is used as an umbrella concept for a diverse array of peer review innovations”. This doesn’t seem to be solving the issue he raises at in beginning. On the contrary considering all peer review innovations as forms of OPR might worsen the current situation. Initiatives like “registered reports”, “shortening review deadlines” or “reviewer recognition”, which are meant to address result-biased peer review, incentivizing reviewers, and boosting review speed, respectively, are certainly peer review innovations but cannot be directly considered as part of the umbrella term of open peer review. Hence the article deserves a stronger conclusion, including perhaps a suggested guideline of clarifying what type of “open peer review” beforehand of any evaluation/discussion using authors seven classifications.”Tony Ross-Hellauer: I thank the reviewer for their time and care in preparing this helpful review report, which has helped to improve the second version. I understand the concern about merely accepting ambiguity to be no way to overcome confusion. I share this view and have now added/altered text to make my intentions clearer - "Given that OPR is such a contested concept, in my view the only sensible way forward is to acknowledge the ambiguity of this term, accepting that it is used as an umbrella concept for a diverse array of peer review innovations. Although it could be argued that merely accepting the status quo in this way does not help resolve possible confusion regarding usage, I would argue that quantifying the ambiguity of usage and mapping the distinct traits enables future discussion to start from a firmer basis that (1) acknowledges that people often mean different things when they use this term, and (2) clarifies in advance exactly which OPR traits are under discussion."BM: “Author can surely strengthen the conclusion by highlighting the lack of evidence in efficiency claims about some of the 7 traits mentioned in the article.”TRH: I have included this point in the expanded conclusion.BM: “I suggest a double checking the term “Post-publication peer review” in WoS search results. Author reports he has used “Open Final version commenting” for this process in making taxonomy reporting only 6 results. However, Post-publication peer review seem to produce a higher number of documents when I searched the term in Scopus results in 55 documents (11 for 2015 and 19 for 2016). This might impact the result reported in Fig. 6 reporting unique configurations of OPR.”TRH: Although there is obvious overlap between the terms "open peer review" and "post-publication peer review", and so the literature on PPPR is useful to understand these phenomena, this study is only an attempt to categorize uses of the former term - hence, articles on PPPR which did not mention OPR  were out of scope.BM: “Although the author reports 22 different definitions there, the figures shows 23 of them.”TRH: The original figure was wrong and has been corrected. The final row of the original figure was a duplicate of the 12th entry . I thank the reviewer very much for their attention to detail in spotting this silly error! Hence, it is correct that there are 22 different definitions.BM: “It is also worth mentioning most of OPR initiatives mentioned in the article are not directly addressing all of shortcomings of the current peer review process. Mainly because one should distinguish between editorial process and peer review process. Issues such as delay and expense are universal to single/double/open peer review as they are part of the editorial process.”TRH: I address this point with added text in the background and conclusion sections. This is an interesting paper addressing a question that is important to journal editors and publishers as well as the wider ‘open science’ community, namely what is meant by open peer review. I have three significant concerns that need to be addressed, followed by more minor annotations and comments. Major issues, in the order they arise in the articleWould this meet most established criteria for a systematic review? Although the author completes a PRISMA checklist, he also notes firstly that he searched Web of Science, then that he added a bunch of other sources, and finally, “This set of articles was further enriched with 42 definitions from sources found through searching for the same terms in other academic databases , Google (for blog articles) and Google Books (for books), as well as following citations in relevant bibliographies and literature reviews." This suggests that it is not really systematic . There is a lack of clarity about the universe of literature that is being assessed, and of details about how it was assessed. In my view, the article is still a worthwhile undertaking, despite being non-systematic, but the title ought to reflect this.In my view the author does not pay enough attention to one important variant of open review, namely real-time review in the open, in which either invited reviewers or ‘the crowd’ comment on an article, with comments being posted as they are ready, rather than at the end of a formal process of peer review and decision-making. Conversely, the inclusion of ‘open platforms’ seems very confusing to me, as they do not have many of the criteria of openness that define the other flavours of open review that the author describes, but instead are about decoupling peer review from publication. Indeed Figure 6 makes plain that as a trait open platforms only occur twice in the 122 definitions considered. I would bet that the ‘real-time’ variant occurs far more often, and ought to be included as one of the key traits within the umbrella definition.In a paper that aims to define open peer review, it is unfortunate that the author doesn’t spend longer considering alternative definitions of peer review. Throughout the article he appears to conflate editorial selection  with technical review (whether the work is sound and properly reported). Thus when he talks about the “problems with peer review” he is sometimes talking about reviewers not spotting technical problems, sometimes about editors rejecting articles that don’t suit their taste, and sometimes about authors going through cycles of editorial rejection to achieve a high impact publication. Conflating these various things does not provide a sound foundation on which to build a definition of open peer review. This conflation is made worse when, for example, it is implied that the only reason for retraction is error. Thus most of the first three paragraphs of the ‘background’ section, and much of what follows about biases, incentives and wastefulness are muddled, and the references and evidence do not all support the broad claims made about ‘peer review’ (itself an ‘umbrella term’).    Abstract and later: The author needs to decide throughout the article whether he is singular (as he appears to be) or uses the royal ‘we’.p7 – it is not helpful to list the seven types of openness without definitions. Even if the lengthy discussion of them follows later, I was desperate for some brief definition, in particular for the last two, and was labouring under a misapprehension about the meaning of the fourth, until much later in the article when I discovered what was meant by ‘open pre-review manuscripts’.  p8 – proponents of open identity review in medicine would also point out that it makes conflicts of interest much more apparent and subject to scrutiny.p8 – some journals use open reports without open identities – i.e. posting reports with published articles but without identifying the reviewer (e.g. http://embor.embopress.org/about#Transparent_Process). The author writes as if open reports must always have open identities.et al. about this issue of terminology (http://biorxiv.org/content/early/2016/12/09/092817)p10 – I think ‘open pre-review manuscripts’ is the wrong name for what the author is describing. At first I thought this meant the practice of posting the authors’ version of a manuscript alongside the peer review history . But I think the author means ‘Open posting before formal review’ . He might like to consider the suggestions by Neylonp10 – I wonder if the author has any evidence that PubPeer has been ‘a major influence’?p11 – I don’t think ‘open platforms’ is the right term either.  In publishing terms a platform is ‘where’ you publish articles, and the author is here discussing an aspect of how you get to the point of publication, and in particular peer review services (which as far as I can tell de facto meet only rather limited criteria of openness). I think what the author is describing is peer review options decoupled from journals  and as noted I don’t understand why these have a place in a definition of open peer review.p11 – Conclusion. I don’t believe the author presents a unified definition of open peer review, for all the reasons discussed above, but he does present most of the traits that together come under the umbrella term. More minor concerns, with page numbers (PDF version) of the corresponding textI have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Theodora Bloom: “This is an interesting paper addressing a question that is important to journal editors and publishers as well as the wider ‘open science’ community, namely what is meant by open peer review. I have three significant concerns that need to be addressed, followed by more minor annotations and comments.”Tony Ross-Hellauer: I'd like to thank the reviewer for their time and care in undertaking this review, which has helped strengthen the paper, especially with regards to the suggestions for strengthening the analysis of the problems with peer review.TB: Would this meet most established criteria for a systematic review? Although the author completes a PRISMA checklist, he also notes firstly that he searched Web of Science, then that he added a bunch of other sources, and finally, “This set of articles was further enriched with 42 definitions from sources found through searching for the same terms in other academic databases , Google (for blog articles) and Google Books (for books), as well as following citations in relevant bibliographies and literature reviews." This suggests that it is not really systematic . There is a lack of clarity about the universe of literature that is being assessed, and of details about how it was assessed. In my view, the article is still a worthwhile undertaking, despite being non-systematic, but the title ought to reflect this.TRH: I have strengthened the description in the methods section to address this criticism. I do believe this article meets the criteria for PRISMA systematic reviews  in the search and date last searched"; and "8. Present full electronic search strategy for at least one database, including any limits used, such that it could be repeated." )TB: “In my view the author does not pay enough attention to one important variant of open review, namely real-time review in the open, in which either invited reviewers or ‘the crowd’ comment on an article, with comments being posted as they are ready, rather than at the end of a formal process of peer review and decision-making. Conversely, the inclusion of ‘open platforms’ seems very confusing to me, as they do not have many of the criteria of openness that define the other flavours of open review that the author describes, but instead are about decoupling peer review from publication. Indeed Figure 6 makes plain that as a trait open platforms only occur twice in the 122 definitions considered. I would bet that the ‘real-time’ variant occurs far more often, and ought to be included as one of the key traits within the umbrella definition.”TRH: I thank the reviewer for pointing out the gap in not mentioning what they term "real-time review in the open" - but I would respectfully disagree with the reviewer that this feature constitutes a core trait. In my view it depends upon other traits, including open reports, commenting, participation and especially open pre-review manuscripts. As this feature depends (in my view) most fully upon open pre-review manuscripts (since the manuscript would need to be online to begin the process), I have included mention of this option in the discussion section for that trait with the added text: "Finally, making manuscripts openly available in advance of review allows comments to be posted as they are received, either from invited reviewers or the wider community, and enabling readers to follow the process of peer-review in real-time."TB: “In a paper that aims to define open peer review, it is unfortunate that the author doesn’t spend longer considering alternative definitions of peer review. Throughout the article he appears to conflate editorial selection  with technical review (whether the work is sound and properly reported). Thus when he talks about the “problems with peer review” he is sometimes talking about reviewers not spotting technical problems, sometimes about editors rejecting articles that don’t suit their taste, and sometimes about authors going through cycles of editorial rejection to achieve a high impact publication. Conflating these various things does not provide a sound foundation on which to build a definition of open peer review. This conflation is made worse when, for example, it is implied that the only reason for retraction is error. Thus most of the first three paragraphs of the ‘background’ section, and much of what follows about biases, incentives and wastefulness are muddled, and the references and evidence do not all support the broad claims made about ‘peer review’ (itself an ‘umbrella term’).”TRH: Since the aim of this article was to clarify, I am very grateful to the reviewer for pointing out the ways in which this section actually confuses things. It was certainly not my intention to insinuate that OPR is a panacea for all problems with traditional peer review. I have added text to address the reviewer's comments: (a) The description of traditional peer review in the Background section has been revised to clarify the role of peer review in scholarly communication; (b) Two new sections have been added to the discussion which make clearer (1) the particular problems with traditional peer review that each OPR trait aims to address, and (2) how each trait can be related to the broader agenda of Open Science .TB: “Abstract and later: The author needs to decide throughout the article whether he is singular (as he appears to be) or uses the royal ‘we’.”TRH: CorrectedTB: “p7 – it is not helpful to list the seven types of openness without definitions. Even if the lengthy discussion of them follows later, I was desperate for some brief definition, in particular for the last two, and was labouring under a misapprehension about the meaning of the fourth, until much later in the article when I discovered what was meant by ‘open pre-review manuscripts’. “TRH: Definitions added.TB: “p8 – proponents of open identity review in medicine would also point out that it makes conflicts of interest much more apparent and subject to scrutiny.”TRH: Added text to address this: "Finally, a reviewer for this paper advises that “proponents of open identity review in medicine would also point out that it makes conflicts of interest much more apparent and subject to scrutiny” . "TB: “p8 – some journals use open reports without open identities – i.e. posting reports with published articles but without identifying the reviewer (e.g. http://embor.embopress.org/about#Transparent_Process). The author writes as if open reports must always have open identities.”TRH: Added sentence to clarify: "Often, although not in all cases , review names are published alongside the reports. "TB: “p10 – I think ‘open pre-review manuscripts’ is the wrong name for what the author is describing. At first I thought this meant the practice of posting the authors’ version of a manuscript alongside the peer review history . But I think the author means ‘Open posting before formal review’ . He might like to consider the suggestions by Neylon et al. about this issue of terminology (http://biorxiv.org/content/early/2016/12/09/092817)”TRH: I agree the terminology could be misconstrued, but am not sure the reviewer's suggestion is preferable (the phrase "posting" could be thought ambiguous) - to avoid ambiguity for future readers, I have added definitions on p7 where I introduce the terms.TB: “p10 – I wonder if the author has any evidence that PubPeer has been ‘a major influence’?”TRH: I’ve weakened the terminology to "An important platform in this regard has been major influence here has been the independent platform Pubpeer "TB: “p11 – I don’t think ‘open platforms’ is the right term either.  In publishing terms a platform is ‘where’ you publish articles, and the author is here discussing an aspect of how you get to the point of publication, and in particular peer review services (which as far as I can tell de facto meet only rather limited criteria of openness). I think what the author is describing is peer review options decoupled from journals  and as noted I don’t understand why these have a place in a definition of open peer review.”TRH: I respectfully disagree with the reviewer here - the word "platform" seems to be used more broadly - e.g., PubPeer considers itself an "online platform for post-publication peer review" - however, I agree that the terminology might be confusing and so have changed the wording to "Open platforms (“decoupled review”)". The point about open platforms being a fringe trait is valid (I've added text to the open platforms section in the discussion to strengthen the acknowledgement of this) - it was included simply because it was observed to be part of the two definitions cited.TB: “p11 – Conclusion. I don’t believe the author presents a unified definition of open peer review, for all the reasons discussed above, but he does present most of the traits that together come under the umbrella term.”TRH: I have removed the term "unified" as this is no doubt contentious General This is a useful, well-written article that helps to clarify some of the “fuzziness” concerning the concept of “Open Peer Review”. The author makes a systematic search of the literature, fully and correctly detailing his methods in the Supplementary Materials. His main conclusion is that  “Open peer review is an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the ethos of Open Science…”. The data from the systematic review fully justifies this conclusion. The references are comprehensive and up to date. On this basis, I believe the quality of the article is already sufficient to justify publication. I would like, nonetheless to suggest some possibilities for improvement.Weak conclusions The author’s conclusions, while correct, are weak. The rapid growth in references to Open Peer Review in the literature suggests that interest in OPR is growing rapidly. It would be useful to point this out. It would also be useful to point out that 110/122 references in his survey talk about “Open Identities” and 72 talk about Open Reports”. This suggests to me that the core sense of Open Peer Review lies precisely in the use of Open identifies and Open Reports and that other aspects are more peripheral. If this were my article (which it is not) I would make this core/periphery distinction more explicit. The author observes correctly that there is still very little evidence about the effectiveness or otherwise of different forms of Open Peer Review. This is another issue that it would be good to bring out in the conclusions.“Power distribution” - The author claims that the configurations of OPR traits “follow a power-law distribution”. Readers will understand what he means. However a power law is a functional relationship between two quantities – and here I see only one (the number of configurations). Power laws play no further part in the author’s argument. So I suggest it would be better to avoid the term. What the author could say, correctly, is that there are a couple of very common configurations and a lot of rarer ones. This links to the idea of a “core” and “peripheral” concepts of OPR.Reasons for open reports The author correctly argues that Open Identities provide an incentive to reviewers to do their work thoroughly. I suggest that the same applies to “Open Reports”. No reviewer wants to expose himself/herself as lazy or blatantly unfair.Detailed points.P4: I suggest the author replaces “Unaccountability” with “Lack of accountability”P6: In the methods, there seem to be two literature surveys, the first by OpenAire (never mentioned again in the rest of the article), the second by the author. The author should clarify exactly who did what and how he used the OpenAire searchP6: The text at the top of column 2 starts in the middle of a sentence. I think something is missing.P7: In Figure 4, it is not clear what is the metric. Is it the number of Journal Articles/Grant proposals etc. or is it the number of distinct definitions found in journals etc? It would be good to clarify what is meant by “Data,Journal Articles”P7: The author writes that “for the literature offers a total of 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature." In reality he found 22 configurations in his, necessarily limited survey. I am certain the literature contains many more. I suggest he corrects his initial statement to make this clear.P7: The definition of “Open identities”, “Open Reports” etc. is given in the discussion. I suggest it would be useful to insert the definitions, earlier, immediately after the introduction of the schema (column 2 p 7)P8: It might be worth mentioning that some publishers (like Frontiers) favor a system of Open Peer Review which publishes reviewers’ names, only when articles are accepted, thereby avoiding the risk of self-censorship by critical reviewers.   I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Richard Walker: “This is a useful, well-written article that helps to clarify some of the “fuzziness” concerning the concept of “Open Peer Review”. The author makes a systematic search of the literature, fully and correctly detailing his methods in the Supplementary Materials. His main conclusion is that “Open peer review is an umbrella term for a number of overlapping ways that peer review models can be adapted in line with the ethos of Open Science…”. The data from the systematic review fully justifies this conclusion. The references are comprehensive and up to date. On this basis, I believe the quality of the article is already sufficient to justify publication. I would like, nonetheless to suggest some possibilities for improvement.”Tony Ross-Hellauer: I'd like to personally thank the reviewer for their time and care in undertaking this review, which has helped strengthen the paper, especially with regards to the suggestions for strengthening the conclusion.RW: Weak conclusions - The author’s conclusions, while correct, are weak. The rapid growth in references to Open Peer Review in the literature suggests that interest in OPR is growing rapidly. It would be useful to point this out. It would also be useful to point out that 110/122 references in his survey talk about “Open Identities” and 72 talk about Open Reports”. This suggests to me that the core sense of Open Peer Review lies precisely in the use of Open identifies and Open Reports and that other aspects are more peripheral. If this were my article (which it is not) I would make this core/periphery distinction more explicit. The author observes correctly that there is still very little evidence about the effectiveness or otherwise of different forms of Open Peer Review. This is another issue that it would be good to bring out in the conclusions.TRH: I agree that the conclusion was weak and welcome the suggestions for improvement. I have written an extended conclusion which I believe addresses all these points.RW: ““Power distribution” - The author claims that the configurations of OPR traits “follow a power-law distribution”. Readers will understand what he means. However a power law is a functional relationship between two quantities – and here I see only one (the number of configurations). Power laws play no further part in the author’s argument. So I suggest it would be better to avoid the term. What the author could say, correctly, is that there are a couple of very common configurations and a lot of rarer ones. This links to the idea of a “core” and “peripheral” concepts of OPR.”TRH: Text changed to "The distribution of traits shows two very popular configurations and a variety of rarer ones ..."RW: “Reasons for open reports - The author correctly argues that Open Identities provide an incentive to reviewers to do their work thoroughly. I suggest that the same applies to “Open Reports”. No reviewer wants to expose himself/herself as lazy or blatantly unfair.”TRH: Agreed - I have added text to this effect: "It could also increase review quality, as the thought of their words being made publicly available could motivate reviewers to be more thorough in their review activities. "RW: “P4: I suggest the author replaces “Unaccountability” with “Lack of accountability””TRH: Text changed as suggested.RW: “P6: In the methods, there seem to be two literature surveys, the first by OpenAire (never mentioned again in the rest of the article), the second by the author. The author should clarify exactly who did what and how he used the OpenAire search”TRH: There was only one lit review, done by the main author as part of the OpenAIRE project - for clarity, I've changed the reference to OpenAIRE to the first person singular ("I")RW: “P6: The text at the top of column 2 starts in the middle of a sentence. I think something is missing.”TRH: I've changed the structure of this sentence to be less confusing: "Sixty-eight percent (n=83) of the 122 definitions identified were explicitly stated, 37.7% (n=46) implicitly stated, and 5.7% (n=7) contained both explicit and implicit information."RW: “P7: In Figure 4, it is not clear what is the metric. Is it the number of Journal Articles/Grant proposals etc. or is it the number of distinct definitions found in journals etc? It would be good to clarify what is meant by “Data,Journal Articles””TRH: This is explained in the text at the bottom of p5 of v1: "Meanwhile, regarding the target of the OPR mentioned in these articles , most were referring to peer review of journal articles (80.7%), with 16% not specifying a target (16%), and a small number of articles also referring to review of data, conference papers and grant proposals."RW: “P7: The author writes that “for the literature offers a total of 22 distinct configurations of seven traits, effectively meaning that there are 22 different definitions of OPR in the literature." In reality he found 22 configurations in his, necessarily limited survey. I am certain the literature contains many more. I suggest he corrects his initial statement to make this clear.”TRH: Added text to sentence to read "in the literature examined here."RW: “P7: The definition of “Open identities”, “Open Reports” etc. is given in the discussion. I suggest it would be useful to insert the definitions, earlier, immediately after the introduction of the schema (column 2 p 7)”TRH: Text added as suggested.RW: “P8: It might be worth mentioning that some publishers (like Frontiers) favor a system of Open Peer Review which publishes reviewers’ names, only when articles are accepted, thereby avoiding the risk of self-censorship by critical reviewers.”TRH: This is a good suggestion.


I hope you are enjoying the new look and feel of the JACMP. The transition to the latest version of software from the Public Knowledge Project was relatively painless, but not completely so. The Editorial Board (myself included) is still getting used to all the new features (and bugs), so please bear with us for the next few months.We still have an open mind about how an open access online journal should function. When we moved to the PKP platform back in 2004, we took advantage of the opportunity for double‐blind peer review. For the most part, that has worked out well, and I have generally had positive feedback about the fairness of the JACMP review process. Moving to the future, one model that may be worth considering is to make the entire review process open access. In this model, de‐identified articles would be posted as soon as received, and anyone would have the opportunity to review the article anonymously for a time (say six weeks). Anyone posting a review would have the opportunity to rate the article. At the end of the review period, the article would be revised by the author, and the revision posted for ranking by the medical physics community. At the end of the ranking period, the top (say twelve) articles would be published and permanently archived. The others would be declined and eliminated from the system. This model eliminates the need for the traditional role of Associate Editors, except perhaps for the need to recruit reviewers if no one is willing to post a review for an article. If you have thoughts on this or other ideas on a beneficial publication model, please send me an email. It is clear to me that we have not seen the end of the changes that open access web technology will force on academia. Thanks for thinking about this.Michael D. Mills, PhD


We are proud to launch a new open access journal, entitled Porcine Health Management (PHM). This new journal, which puts the pig in a central position, has been jointly established by the European College of Porcine Health and Management and the European Association of Porcine Health Management.PHM is interested in publishing articles of high quality and novelty, but we also consider research results of more general interest and clinical features to the readers. We believe that putting the pig at the center of this journal is unique and will serve both our favored animal species and our professions.We hope that you will enjoy the PHM journal and that you will consider it when deciding where to publish your work. Pigs are so compelling, so mysterious, so contradictory – finicky yet fat, massive yet dainty, stolid yet smart. We can’t decide what we think of them. We’re uneasy about our role in their life and death. We make images of them that reveal our complex feelings—our affection, our revulsion, our sentimentality, and our guilt” .Harry Truman said “No man should be allowed to be president who does not understand hogs” .“Dogs look up to you, cats look down on you and pigs think you’re their equal” is attributed to an old rural New England hogophile .George Orwell in his book “Animal Farm” mentions the natural hierarchy that evolves when animals live together: “the work of teaching and organizing the others naturally fell to the pigs, who were recognized as being the cleverest of animals”.In many aspects, the physiology of pigs resembles very closely that of humans, and pigs often serve as experimental models for surgery, for performing physiological studies, and also for studying pathogenesis and immunological responses to a variety of infectious organisms.It is well known that, for ages and throughout the globe, pigs have intrigued humans and have even been a close part of their life. For this reason, the relationship between people and pigs has frequently been pictured by politicians, cartoonists, and writers .It is also clear that the pig, as animal species, has been and still is being studied intensively from different angles by medical researchers, geneticists, nutritionists, and other porcine specialists. Intensification and industrialization of pig production in recent times has been challenging not only for the pig itself but also for those of us who are directly involved. Every aspect of pig keeping has become highly specialized and demanding; new diseases emerge, changes occur rapidly and are sometimes hard to follow. With this rapid evolution, the whole pig, the entire picture of pig raising, the interdisciplinary features, the interactive elements involved and their effects are often difficult to follow, and this process may even become more problematic in the future.Let us admit, the pig is an amazing animal that has served the human population in different ways, in good and bad times and throughout history, and still continues to do so. This animal species thus asks for and deserves a science-based journal that is concerned with various aspects related to medicine and production in modern pig raising and that puts the pig centrally on the scene.Clearly, it is rather difficult nowadays for the pig scientist involved in different pig-related subjects, for the pig veterinarian in his everyday job and for the pig manager, to keep up with all the scientific information published in different sources of the literature.Porcine Health Management (PHM) . This new journal, which puts the pig in a central position, has been jointly established by the European College of Porcine Health and Management (http://www.ecphm.org) and the European Association Porcine of Porcine Health Management (http://www.eaphm.org).We are, therefore, proud to launch a new open access journal, entitled PHM intends to treat different topics of swine raising including: infectious and non-infectious diseases, reproduction, epidemiology, housing, nutrition, medicines, management, economics, genetics, animal welfare, ethics, legislation, and food safety. It aims to serve as an international platform for sharing new and original research findings, results of clinical studies, presentation of new and interesting clinical cases and recent evolutions. The journal will publish this information as up-to-date research articles, topical review articles, short communications, reports on interesting cases, as well as clinical studies commentaries. The foremost goal of this journal is thus to provide a platform to bridge the knowledge gap between physiological and pathological aspects of pig raising, to report on new developments and on important production management features, all within a science-based framework. This approach is unique at present in the porcine field worldwide.We are interested in publishing articles of high quality and novelty, but we also consider research results of more general interest and clinical features to the readers. We believe that putting the pig at the center of this journal is unique and will serve both our favored animal species and our professions.http://www.porcinehealthmanagement.com/authors/instructions), all articles published in PHM by BioMed Central [Being open access ( Central  are freeWe believe that the open access format offers many advantages relative to traditional publication strategies, as articles will be published online very shortly after editorial acceptance which, markedly decreasing the time for distribution. All manuscripts will be independently reviewed by at least 2 experts in the field and by an Editor-in-Chief. The journal benefits from an editorial board of over 30 highly specialized members from 16 different countries. It is our aim to deliver a timely and fair peer review process. Authors of articles published in PHM retain the copyright of their articles and are free to reproduce and disseminate their work.Among PHM’s inaugural articles are a review article on tail biting in modern pig production by Anna Valros and Mari Heinonen , a reseaWe hope that you will enjoy the PHM journal and that you will consider it when deciding where to publish your work.http://www.porcinehealthmanagement.com/my/preferences to receive new article alerts from Porcine Health Management.Sign up at


Research Involvement and Engagement. The journal considers manuscripts on any aspect of the engagement or involvement of patients, carers or members of the public in design, delivery and dissemination of research in health or social care.This editorial introduces the new, online, open-access journal We support the involvement of patients in the production of the journal, as co-authors, as reviewers and as consumers of the journal, through plain English summaries. We want patients and the wider public to become engaged with research and influence its future direction. Through the co-production of the Journal, we wish to encapsulate the vision of “nothing about me without me”.Research Involvement and Engagement provides a dedicated place for publication of research focused on public involvement and engagement, as well as a forum for discussion of methodological issues that will lead to increased scientific rigour in public involvement and to an enhanced evidence base to underpin best practice internationally.The journal of  Research Involvement and Engagement. This is a bold and exciting addition to the BioMed Central portfolio of journals. Research Involvement and Engagement (or RIE) is an interdisciplinary health and social care journal, focussing on patient and public involvement and engagement in research at all stages, including its production, dissemination and implementation. We publish empirical papers and analyses of patient and public involvement and public engagement with research in health and social care, opinion pieces and methodology articles. We encourage critical reflection which enhances our understanding of involvement and engagement. The launch of RIE is very timely and reflects the significant increase in interest in patient and public involvement in research internationally and the need to report public involvement in papers, to generate a sound and robust evidence base from which others can draw to develop best practice.Welcome to this first edition of a new journal called RIE is a new type of journal focused on developing a partnership between the public and researchers. We believe collaborative involvement between the public and researchers plays an important role in developing high-quality research, ensuring it is relevant, acceptable and appropriate for patients and the public [e public . Such ine public .Active forms of involvement in research reflect a fundamental paradigm shift in health and social care research, away from paternalism towards partnership, with members of the public collaboratively involved throughout the research process, including as authors and co-authors of papers. This change in the nature of research reflects an international phenomenon, with many countries encouraging the development of closer collaboration between the public and researchers, exemplified by work of groups such as the Health Technology International Patient and Citizen Involvement Group .RIE will provide an important conduit for the development of the involvement evidence base, which in many ways, it still in its infancy. We hope to contribute to its maturation and greater recognition internationally. Strategic reviews, such as the “Breaking Boundaries Review” commissioned by the Director General Research and Development/Chief Medical Officer (CMO) Department of Health, England, have strengthened the vision of involvement as “a population actively involved in research to improve health and wellbeing for themselves, their family and their communities”. In this future scenario, involvement will become second nature and research without evidence of public involvement would be considered flawed [RIE contributes to this paradigm shift by helping to build the evidence base that will contribute to more effective practice. With this mind, we are particularly keen to receive international submissions from countries at the start of their involvement and engagement journey.While much progress has been made, there is still a long way to go before active involvement and co-production of knowledge become the norm in health research internationally. We hope d flawed . We see RIE has been established. RIE is online and open-access, to ensure that it is available to patients, to the public and to researchers alike, anywhere in the world. Open-access publishing offers the opportunity for widening access to papers, to a deliberative discussion internationally, helping nurture the partnership between the public and researchers. The journal will be co-produced by all key stakeholders, including patients, policymakers, academics and funders. RIE has a unique governance structure, which includes patients on the editorial board and a patient as one of the two joint editors. RIE operates using an open peer review system, where the reviewers’ names are included on the peer review reports. In addition, if the article is published, the named reports are published online alongside the article as part of a “pre-publication history”. All previous versions of the manuscript and all author responses to the reviewers are also available to readers.We have tried to reflect our values and ways of working in the way that Moreover, because of our editorial commitment to co-production on knowledge and the multi-disciplinary nature of the range of topics we can cover, patients, members of the public and researchers will benefit from having the journal as a shared resource for exchanging ideas and experiences, as well as outcomes and findings.RIE are made freely and permanently accessible online immediately upon publication, without subscription charges or registration barriers. It is a fundamental principle of this journal that the knowledge we gain from patient involvement and engagement should made available to all, freely, including of course patients and the public.All articles published by RIE are the copyright holders of their articles and have granted to any third party, in advance and in perpetuity, the right to use, reproduce or disseminate the article, according to the BioMed Central copyright and license agreement [Authors of articles published in greement .RIE pipeline.We are excited to publish the initial set of articles and commentaries which cover a broad range of areas of importance in patient and public. The key paper by Crowe and colleagues demonstrates the importance of including the public in the prioritisation of research , while MResearch Involvement and Engagement. We invite you to submit your papers to Research Involvement and Engagement, the first journal anywhere in the world to specialise in patient and public involvement and engagement.We are very excited about the launch of