Peer review of research articles is a core part of our scholarly communication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communications infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These developments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement incentives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possible. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments. Peer review is a core part of our self-regulating global scholarship system. It defines the process in which professional experts (peers) are invited to critically assess the quality, novelty, theoretical and empirical validity, and potential impact of research by others, typically while it is in the form of a manuscript for an article, conference, or book . For thePeer review is not a singular or static entity. It comes in various flavors that result from different approaches to the relative timing of the review in the publication cycle, the reciprocal transparency of the process, and the contrasting and disciplinary practices . Such inPLOS ONE,PeerJ, theFrontiers series). Such a flexibility in the filter function of peer review reduces, but does not eliminate, the role of peer review as a selective gatekeeper, and can be considered to be “impact neutral.” Due to such digital experimentations, ongoing discussions about peer review are intimately linked with contemporaneous developments in Open Access (OA) publishing and to broader changes in open scholarship  as well as specialized research databases  andPubMed). Particularly relevant articles were used to seed identification of cited, citing, and articles related by citation. The team co-ordinated efforts using an online collaboration tool (Slack) to share, discuss, debate, and come to consensus. Authoring and editing was also done collaboratively and in public view usingOverleaf. Each co-author independently contributed original content and participated in the reviewing, editing and discussion process.This article provides a general review of conventional journal article peer review and evaluation of recent and current innovations in the field. It is not a systematic review or meta-analysis of the empirical literature . Rather, a team of researchers with diverse expertise in the sciences, scholarly publishing and communication, and libraries pooled their knowledge to collaboratively and iteratively analyze and report on the present literature and current innovations. The reviewed and cited articles within were identified and selected through searches of general research databases .keeping . Researckeeping . EvaluatThis editor-led process of peer review became increasingly mainstream and important in the post-World War II decades, and is what we term “traditional” or “conventional” peer review throughout this article. Such expansion was primarily due to the development of a modern academic prestige economy based on the perception of quality or excellence surrounding journal-based publications . The incThe result of this is that modern peer review has become enormously complicated. By allowing the process to become managed by a hyper-competitive publishing industry and integrated with academic career progression, developments in scholarly communication have become strongly coupled to the transforming nature of academic research institutes. These institutes have now evolved into internationally competitive businesses that strive for impact through journal publication. Often this is now mediated by commercial publishers through attempts to align their products with the academic ideal of research excellence . Here, e1.1.3 The peer review revolution. In the last several decades, and boosted by the emergence of Web-based technologies, there have been substantial innovative efforts to decouple peer review from the publishing process (arXiv (arxiv.org) formalized this process by creating a central network for whole communities to access such e-prints. Today,arXiv has more than one million e-prints from various research fields and receives more than 8,000 monthly submissions  in 2001 offered a step towards bringing journals and peer review back to their community-led roots, by providing the technology to implement a range of potential peer review models within a low-cost open source platform. As of 2015, the OJS platform provided the technical infrastructure and editorial and peer review workflow management support to more than 10,000 journals , that called for systemic changes in the way that scientific research outputs are evaluated, and advances in Web-based technologies, are likely catalysts for such innovation. Born-digital journals, such as thePLOS series, introduced commenting on published papers, and Rapid Responses byBMJ has been highly successful in providing a platform for formalised comments (bmj.com/rapid-responses). Such initiatives spurred developments in cross-publisher annotation platforms likePubPeer (pubpeer.com/) andPaperHive (paperhive.org/). Some journals, such asF1000 Research (f1000research.com/) andThe Winnower (thewinnower.com/), rely exclusively on a model where peer review is conducted after the manuscripts are made publicly available. Other services, such asPublons (publons.com/), enable reviewers to claim recognition for their activities as referees. Originally,Academic Karma (academickarma.org/) offered a similar service toPublons, but has since adapted its model to facilitate peer review of preprints. Platforms such asScienceOpen (scienceopen.com/) provide a search engine combined with peer review across publishers on all documents, regardless of whether manuscripts have been previously reviewed. Each of these innovations has partial parallels to other social Web applications or platforms in terms of transparency, reputation, performance assessment, and community engagement. It remains to be seen whether these new models of evaluation will become more popular than traditional peer review, either singularly or in combination.The past five to ten years have seen an accelerating wave of innovation in peer review, which we term “the revolution” phase . The economic motivations for continuing to impose selectivity in a digital environment, and applying peer review as a mechanism for this, have received limited attention or questioning, and are often simply regarded as how things are done. Use of selectivity is now often attributed to quality control, but may be more about building the brand and the demand from specific publishers or venues. Proprietary reviewer databases that enable high selectivity are seen as a good business asset. In fact, the attribution is based on the false assumption that peer review requires careful selection of specific reviewers to assure a definitive level of adequate quality, termed the “Fallacy of Misplaced Focus” byIn addition to being used to judge submitted material for acceptance at a journal, review comments provided to the authors serve to improve the work and the writing and analysis skills of the authors. This feedback can lead to improvements to the submitted work that are iterated between the authors, reviewers, and editor, until the work is either accepted or the editor decides that it cannot be made acceptable for their specific scientific journal. In other cases, it allows the authors to improve their work to prepare for a new submission to another venue. In both cases, a good  peer review should provide general feedback that allows authors to improve their skills and competency at preparing and presenting their research. In a sense, good peer review can serve as distributed mentorship.The Philosophical Transactions of the Royal Society—to the invention of peer review. The two are often regarded to be coupled by necessity, largely ignoring the complex and interwoven histories of peer review and publishing. This has consequences, as the individual identity of a scholar is strongly tied to specific forms of publication that are evaluated in particular ways  have been important to improve the reporting of research and its peer review according to standardised criteria. Another response to issues with scholarly publishing has been COPE, the Committee on Publication Ethics (publicationethics.org), established in 1997 to address potential cases of abuse and misconduct during the publication process , and later created specific guidelines for peer review. Yet, the effectiveness of this initiative at a system-level remains unclear. A popular and widely-cited editorial inThe BMJ made some quite serious allegations at peer review, stating that it is “slow, expensive, profligate of academic time, highly subjective, prone to bias, easily abused, poor at detecting gross defects, and almost useless at detecting fraud” , and which have since become incredibly successful ventures  systems are rapidly emerging, and innovation has been accelerating over the last several years ;Table 3However, the context of this transparency and the implications of different modes of transparency at different stages of the review process are both very rarely explored. Progress towards achieving transparency has been variable but generally slow across the publishing system. Engagement with experimental open models is still far from common, in part perhaps due to a lack of rigorous evaluation and empirical demonstration that they are more effective processes. A consequence of this is the entrenchment of the ubiquitously practiced and much more favored traditional model . However, as history shows, such a process is non-traditional but nonetheless currently held in high regard. Practices such as self-publishing and predatory or deceptive publishing cast a shadow of doubt on the validity of research posted openly online that follow these models, including those with traditional scholarly imprints .OpenAIRE found 122 different definitions of OPR in use, exemplifying the extent of this issue. This diversity was distilled into a single proposed definition comprising seven different traits of OPR: participation, identity, reports, interaction, platforms, pre-review manuscripts, and final-version commenting  can be sorted into four main categories, which are examined in more detail below. Each of these feed into the wider core issues in peer review of incentivizing engagement, providing appropriate recognition and certification, and quality control and moderation:1.How can referees receive credit or recognition for their work, and what form should this take;2.Should referee reports be published alongside manuscripts;3.Should referees remain anonymous or have their identities disclosed;4.Should peer review occur prior or subsequent to the publication process .A vast majority of researchers see peer review as an integral and fundamental part of their workIncrease the total pool of potential referees,Editorial staff more thoroughly vet submissions prior to sending for review,Increase acceptance rates to avoid review duplication,Impose a production cap on authors,Decrease the number of referees per paper, and/orDecrease the time spent on peer review.Of these, the latter two can both potentially reduce the quality of peer review and therefore affect the overall quality of published research. Paradoxically, while the Web empowers us to communicate information virtually instantaneously, the turn around time for peer reviewed publications remains quite long by comparison. One potential solution is to encourage referees by providing additional recognition and credit for their work. The present lack of bona fide incentives for referees is perhaps one of the main factors responsible for indifference to editorial outcomes, which ultimately leads to the increased proliferation of low quality research  andCollabra (collabra.org/about/our-model), published by University of California Press, as well as most statistical referees (www.jclinepi.com/article/S0895-4356(16)30707-7/fulltext). Another idea that journals and publishers have tried implementing is to list the best reviewers for their journal , or, on the basis of a suggestion byDigital Medievalist from 2005–2016; seebit.ly/DigitalMedievalistArchive for examples preserved in the Internet Archive).Digital Medievalist stopped using this model and removed the colophon as part of its move to theOpen Library of Humanities; cf.journal.digitalmedievalist.org). As such, authors can then integrate this into their scholarly profiles in order to differentiate themselves from other researchers or referees. Currently, peer review is poorly acknowledged by practically all research assessment bodies, institutions, granting agencies, as well as publishers, in the process of professional advancement or evaluation. Instead, it is viewed as expected or normal behaviour for all researchers to contribute in some form to peer review.eferees . Other j2.2.2 Increasing demand for recognition. These traditional approaches of credit fall short of any sort of systematic feedback or recognition, such as that granted through publications. A change here is clearly required for the wealth of currently unrewarded time and effort given to peer review by academics. A recent survey of nearly 3,000 peer reviewers by the large publisherWiley showed that feedback and acknowledgement for work as referees are valued far above either cash reimbursements or payment in kind  provides a stable means of integrating these platforms with persistent researcher identifiers in order to receive due credit for reviews.ORCID is rapidly becoming part of the critical infrastructure for open OPR, and greater shifts towards open scholarship  service  was designed to breathe some transparency into peer review, and provide information about the peer review itself without exposing the reports . While it describes itself as a service to identify fraud and maintain the integrity of peer review, it remains unclear whether it has achieved these objectives in light of the ongoing criticisms of the conventional process.The rationale behind publishing referee reports lies in providing increased context and transparency to the peer review process, and can occur irrespective of whether or not the reviewers reveal their identities. Often, valuable insights are shared in reviews that would otherwise remain hidden if not published. By publishing reports, peer review has the potential to become a supportive and collaborative process that is viewed more as an ongoing dialogue between groups of scientists to progressively assess the quality of research. Furthermore, the reviews themselves are opened up for analysis and inspection, including how authors respond to reviews, adding an additional layer of quality control and a means for accountability and verification. There are additional educational benefits to publishing peer reviews, such as training purposes or for journal clubs. Given the inconclusive evidence regarding the training of referees . ThePeeIn a study of two journals, one where reports were not published and another where they were,Assessments of research articles can never be evidence-based without the verification enabled by publication of referee reports. However, they are still almost ubiquitously regarded as having an authoritative, and uniform, stamp of quality. The issue here is that the attainment of peer reviewed status will always be based on an undefined, and only ever relative, quality threshold due to the opacity of the process. This is in itself quite an unscientific practice, and instead, researchers rely almost entirely on heuristics and trust for a concealed process and the intrinsic reputation of the journal, rather than anything legitimate. This can ultimately result in what is termed the “Fallacy of Misplaced Finality”, described byPublishing peer review reports appears to have little or no impact on the overall process but may encourage more civility from referees. In a small survey,BioMed Central launched in 2000, it quickly recognized the value in including both the reviewers’ names and the peer review history (pre-publication) alongside published manuscripts in their medical journals in order to increase the quality and value of the process. Since then, further reflections on OPR , launched in early 2016.There are different levels of bi-directional anonymity throughout the peer review process, including whether or not the referees know who the authors are but not vice versa , or wheWhile there is much potential value in anonymity, the corollary is also problematic in that anonymity can lead to reviewers being more aggressive, biased, negligent, orthodox, entitled, and politicized in their language and evaluation, as they have no fear of negative consequences for their actions other than from the editor. . In theoOn the other hand, eponymous peer review has the potential to inject responsibility into the system by encouraging increased civility, accountability, declaration of biases and conflicts of interest, and more thoughtful reviews . However2.4.1 Reviewing the evidence. Reviewer anonymity can be difficult to protect, as there are ways in which identities can be revealed, albeit non-maliciously. For example, through language and phrasing, prior knowledge of the research and a specific angle being taken, previous presentation at a conference, or even simple Web-based searches.A randomized trial showed that blinding reviewers to the identity of authors improved the quality of the reviews , they will become targets for retaliatory backlashes from more senior researchers , although this is a relatively recent program and the effectiveness of it can not yet be assessed.In an ideal world, we would expect that strong, honest, and constructive feedback is well received by authors, no matter their career stage. Yet, there seems to be the very real perception that this is not the case. Retaliations to referees in such a negative manner can represent serious cases of academic misconduct . It is i2.4.3 The impact of identification and anonymity on bias. One of the biggest criticisms levied at peer review is that, like many human endeavours, it is intrinsically biased and not the objective and impartial process many regard it to be. Yet, the question is no longer about whether or not it is biased, but to what extent it is in different social dimensions - a debate which is very much ongoing . One of , and a persistent academic view that published equals peer reviewed. Numerous venues now provide inbuilt systems for post-publication peer review, includingRIO,PubPub,ScienceOpen,The Winnower, andF1000 Research. Some European Geophysical Union journals hosted onCopernicus offer a hybrid model with initial discussion papers receiving open peer review and comments and then selected papers accepted as final publications, which they term ‘Interactive Public Peer Review’ (publications.copernicus.org/services/public_peer_review.html). Here, review reports are posted alongside published manuscripts, with an option for reviewers to reveal their identity should they wish  from peer review and the certification that this provides. Some even regard this decoupling process as the “paradigm shift” that scholarly publishing needs . Some puey wish . In addiPeerage of Science(peerageofscience.org),RUBRIQ  (rubriq.com), andAxios Review  have implemented decoupled models of peer review. These tools work based on the same core principles as traditional peer review, but authors submit their manuscripts to the platforms first instead of journals. The platforms provide the referees, either via subject-specific editors or via self-managed agreements. After the referees have provided their comments and the manuscript has been improved, the platform forwards the manuscript and the referee reports to a journal. Some journal policies accept the platform reviews as if the reviews were coming from the journal’s pool of reviewers, while others still require the journal’s handling editor to look for additional reviewers. While these systems usually cost money for authors, these costs can sometimes be deducted from any publication fees once the article has been published. Journals accept deduction of these costs because they benefit by receiving manuscripts that have already been assessed for journal fit and have been through a round of revisions, thereby reducing their workload. A consortium of publishers and commercial vendors recently established the Manuscript Exchange Common Approach  as a form of portable review in order to cut down inefficiency and redundancy. Yet, it still is in too early a stage to comment on its viability.Initiatives such as theLIBRE (openscholar.org.uk/libre) is a free, multidisciplinary, digital article repository for formal publication and community-based evaluation. Reviewers’ assessments, citation indices, community ratings, and usage statistics, are used byLIBRE to calculate multiparametric performance metrics. At any time, authors can upload an improved version of their article or decide to send it to an academic journal. Launched in 2013,LIBRE was subsequently combined with theSelf-Journal of Science (sjscience.org) under the combined heading ofOpen Scholar (openscholar.org.uk). One of the tools thatOpen Scholar offers is a peer review module for integration with institutional repositories, which is designed to bring research evaluation back into the hands of research communities themselves (openscholar.org.uk/open-peer-review-module-for-repositories/).Academic Karma is another new service that facilitates peer review of preprints from a range of sources (academickarma.org/).2.5.1 Preprints and overlay journals. In fields such as mathematics, astrophysics, or cosmology, research communities already commonly publish their work on thearXiv platform  andpsyarXiv (psyarxiv.com). Social movements such as ASAPBio (asapbio.org) are helping to drive this expansion. Manuscripts submitted to these preprint servers are typically a draft version prior to formal submission to a journal for peer review, but can also be updated to include peer reviewed versions . Primary motivation here is to bypass the lengthy time taken for peer review and formal publication, which means the timing of peer review occurs subsequent to manuscripts being made public. However, sometimes these articles are not submitted anywhere else and form what some regard as grey literature (Naturenature.com/nature/authors/gta/#a5.4),Cellcell.com/cell/authors), and recently the scholarly publishing sector even attempted to discredit their recognition as valuable publications (asapbio.org/faseb). In spite of this, the popularity and success of preprints is testified by their citation records, with four of the top five venues in physics and maths beingarXiv sub-sections (scholar.google.com/citations?view_op=top_ venues&hl=en&vq=phy). Similarly, the single most highly cited venue in economics is theNBER Working Papers  server (scholar.google.com/citations?view_op=top_venues&hl=en&vq=bus_economics), according to theGoogle Scholar h5-index. extent . In otheerature . Papers erature . Some joThe Open Journal (theoj.org) orDiscrete Analysis  are exclusively peer review platforms that circumvent traditional publishing by utilizing the pre-existing infrastructure and content of preprint servers likearXiv. Peer review is performed easily, rapidly, and cheaply, after initial publication of the articles. The reason they are termed “overlay” journals is that the articles remain onarXiv in their peer reviewed state, with the “journals” mostly comprising a simple list of links to these versions , which allows authors to self-publish their work.PubPub then provides a mechanism for creating overlay journals that can draw from and curate the content hosted on the platform itself. This model incorporates the preprint server and final article publishing into one contained system.EPISCIENCES is another platform that facilitates the creation of peer reviewed journals, with their content hosted on digital repositories . Editors compile articles to form a collection, write an editorial, and can invite referees to peer review the articles. This process is automatically mediated byORCID for quality control , andCrossRef andCreative Commons licensing for appropriate recognition. They are essentially equivalent to community-mediated overlay journals, but with the difference that they also draw on additional sources beyond preprints.A similar approach to that of overlay journals is being developed by2.5.2 Two-stage peer review and Registered Reports. Registered Reports represent a significant departure from conventional peer review in terms of relative timing and increased rigour . Registered Reports are designed to boost research integrity by ensuring the publication of all research results, which helps reduce publication bias. As opposed to the traditional model of publication, where “positive” results are more likely to be published, results remain unknown at the time of the first review stage and therefore even “negative” results are equally as likely to be published. Such a process is designed to incentivize data-sharing, guard against dubious practices such as selective reporting of results  and low statistical power, and also prioritizes accurate reporting over that which is perceived to be of higher impact or publisher worthiness. rigour . Here, p rigour . Such a ournals  . This model requires an author to solicit reviews from their peers prior to submission in order to assess the suitability of a manuscript for publication. While some might see this as a potential bias, it is worth bearing in mind that many journals already ask authors who they want to review their papers, or who they should exclude. To avoid potential pre-submission bias, reviewer identities and their endorsements are made publicly available alongside manuscripts, which also removes any possible deleterious editorial criteria from inhibiting the publication of research. Also, PRE has been suggested by Jan Velterop to be much cheaper, legitimate, unbiased, faster, and more efficient alternative to the traditional publisher-mediated method . PRE also has the potential advantage of being more useful to non-native English speaking authors by allowing them to work with editors and reviewers in their first languages. However, possible drawbacks of this process include positive bias imposed by having author-recommended reviewers, as well as the potential for abuse through suggesting fake reviewers. As such, such a system highlights the crucial role of an Editor for verification and mediation.2.5.4 Limitations of decoupled peer review. Despite a general appeal for post-publication peer review and considerable innovation in this field, the appetite among researchers is limited, reflecting an overall lack of engagement with the process , which hinders its recognition as a valuable community process and for additional evaluation or assessment decisions.As recently as 2012, it was reported that relatively few platforms allowed users to evaluate manuscripts post-publication . Even plsee also. Part ofarXiv spin-offs likebiorXiv, and potential layered peer review through services such as the recently launchedPeer Community In (peercommunityin.org).While several new overlay journals are currently thriving, the history of their success is invariably limited, and most journals that experimented with the model returned to their traditional coupled roots . FinallyAxios Review was closed down in early 2017 due to a lack of uptake from researchers, with the founder stating: “I blame the lack of uptake on a deep inertia in the researcher community in adopting new workflows”  is an open-source, community-based platform where users submit comments and original or linked content, organized into thematic lists of subreddits. AsReddit, highly scoring submissions are relatively ephemeral, with an automatic down-voting algorithm implemented that shifts them further down lists as new content is added, typically within 24 hours of initial posting.3.1.1 Reddit as an existing “journal” of science. The subreddit for Science (reddit.com/r/science) is a highly-moderated discussion channel, curated by at least 600 professional researchers and with more than 15 million subscribers at the time of writing. The forum has even been described as “The world’s largest 2-way dialogue between scientists and the public” (public” . ContribReddit through science AMAs (Ask Me Anythings), which tend to be quite popular. However, the level of discourse provided in this is generally not equivalent in depth compared to that perceived for peer review, and is more akin to a form of science communication or public engagement with research. In this way,Reddit has the potential to drive enormous amounts of traffic to primary research and there even is a phenomenon known as the “Reddit hug of death”, whereby servers become overloaded and crash due toReddit-based traffic. The/r/science subreddit is viewed as a venue for “scientists and lay audiences to openly discuss scientific ideas in a civilized and educational manner”, according to the organizer, Dr. Nathan Allen  by the community. All links or texts can be publicly discussed in terms of methods, context, and implications, similar to any scholarly post-publication commenting system. Such a process for peer review could essentially operate as an additional layer on top of a preprint archive or repository, much like a social version of an overlay journal. Ultimately, a public commenting system like this could achieve the same depth of peer evaluation as the formal process, but as a crowd-sourced process. However, it is important to note here that this is a mode of instantaneous publication prior to peer review, with filtering through interaction occurring post-publication. Furthermore, comments can receive similar treatment to submitted content, in that they can be upvoted, downvoted, and further commented upon in a cascading process. An advantage of this is that multiple comment threads can form on single posts and viewers can track individual discussions. Here, the highest-ranked comments could simply be presented at the top of the thread, while those of lowest ranking remain at the bottom.Reddit allows any user to create their own subreddit, pending certain status achievements through platform engagement. In addition, this could be moderated externally throughORCID, where a set number of published items in an ORCID profile are required for that individual to perform a peer review; or in this case, create a new subreddit. Connection to an academic profile within academia, such asORCID, further allows community validation, verification, and judgement of importance. For example, being able to see whether senior figures in a given field have read or upvoted certain threads can be highly influential in decisions to engage with that thread, and vice versa. A very similar process already occurs at theSelf Journal of Science (sjscience.org/), where contributors have a choice of voting either “This article has reached scientific standards” or “This article still needs revisions”, with public disclosure of who has voted in either direction. Threaded commenting could also be implemented, as it is vital to the success of any collaborative filtering platform, and also provides a highly efficient corrective mechanism. Peer evaluation in this form emphasizes progress and research as a discourse over piecemeal publications or objects as part of a lengthier process. Such a system could be applied to other forms of scientific work, which includes code, data and images, thereby allowing contributors to claim credit for their full range of research outputs. Comments could be signed by default, pseudonymous, or anonymized until a contributor chooses to reveal their identity. If required, anonymized comments could be filtered out automatically by users. A key to this could be peer identity verification, which can be done at the back-end via email or integrated viaORCID.In theory, a subreddit could be created for any sub-topic within research, and a simple nested hierarchical taxonomy could make this as precise or broad as warranted by individual communities.3.1.3 Translating engagement into prestige.Reddit karma points are awarded for sharing links and comments, and having these upvoted or downvoted by other registered members. The simplest implementation of such a voting system for peer review would be through interaction with any article in the database with a single click. This form of field-specific social recommendation for content simultaneously creates both a filter and a structured feed, similar toFacebook andGoogle+, and can easily be automated. With this, contributions get a rating, which accumulate to form a peer-based rating as a form of reputation and could be translated into a quantified level of community-granted prestige. Ratings are transparent and contributions and their ratings can be viewed on a public profile page. More sophisticated approaches could include graded ratings—e.g., five-point responses, like those used byAmazon—or separate rating dimensions providing peers with an immediate snapshot of the strengths and weaknesses of each article. Such a system is already in place atScienceOpen, where referees evaluate an article for each of its importance, validity, completeness, and comprehensibility using a five-star system. For any given set of articles retrieved from the database, a ranking algorithm could be used to dynamically order articles on the basis of a combination of quality , relevance (using a recommendation system akin toAmazon), and recency (newly added articles could receive a boost). By default, the same algorithm would be implemented for all peers, as onReddit. The issue here is making any such karma points equivalent to the amount of effort required to obtain them, and also ensuring that they are valued by the broader research community and assessment bodies. This could be facilitated through a simple badge incentive system, such as that designed by theCenter for Open Science for core open practices (cos.io/our-services/open-science-badges/).3.1.4 Can the wisdom of crowds work with peer review? One might consider aReddit-style model as pitching quantity versus quality. Typically, comments provided onReddit are not at the same level in terms of depth and rigor as those that we would expect from traditional peer review—as in, there is more to research evaluation than simply upvoting or downvoting. Furthermore, the range of expertise is highly variable due to the inclusion of specialists and non-specialists as equals (“peers”) within a single thread. However, there is no reason why a user prestige system akin toReddit flair cannot be utilised to differentiate varying levels of expertise. The primary advantage here is that the number of participants is uncapped, therefore emphasizing the potential thatReddit has in scaling up participation in peer review. With aReddit model, we must hold faith that sheer numbers will be sufficient in providing an optimal assessment of any given contribution and that any such assessment will ultimately provide a consensus of high quality and reusable results. Social review of this sort must therefore consider at what point is the process of review constrained in order to produce such a consensus, and one that is not self-selective as a factor of engagement rather than accuracy. This is termed the “Principle of Multiple Magnifications” byF1000 Research already have such a tagging system in place, where reviewers can mark a submission as approved after successive peer review iterations.“The rich get richer” is one potential phenomenon for this style of system. Content from more prominent researchers may receive relatively more comments and ratings, and ultimately hype, as with any hierarchical system, including that for traditional scholarly publishing. Research from unknown authors may go relatively under-noticed and under-used, but will at least have been publicized. One solution to this is having a core community of editors, drawing on the r/science subreddit’s community of moderators. The editors could be empowered to invite peers to contribute to discussion threads, essentially wielding the same executive power as a journal editor, but combined with that of a forum moderator. Recent evidence suggests that such intelligent crowd reviewing has the potential to be an efficient and high quality process .Amazon (amazon.com/) was one of the first websites allowing the posting of public customer book reviews. The process is completely open to participation and informal, so that anyone can write a review and vote, providing usually that they have purchased the product. Customer reviews of this sort are peer-generated product evaluations hosted on a third-party website, such asAmazon .Amazon . Here, u3.2.1 Amazon’s star-rating system. Star-rating systems are used frequently at a high-level in academia, and are commonly used to define research excellence, albeit perhaps in a flawed and an arguably detrimental way; e.g., the Research Excellence Framework in the UK (ref.ac.uk)  indicates an extremely negative view, whereas a high rating (five stars) reflects a positive view of the product. An intermediate scoring (three stars) can either represent a mid-view of a balance between negative and positive points, or merely reflect a nonchalant attitude towards a product. These ratings reveal fundamental details of accountability and are a sign of popularity and quality for items and sellers..ac.uk) . A studyYouTube, for instance, is five stars and it turns out that this is repeated across the Web with an overall average estimated at about 4.3 stars, no matter the object being rated . Reputation systems intend to achieve three things: foster good behavior, penalize bad behavior, and reduce the risk of harm to others as a result of bad behavior (ORCID). One further problem with reputation systems is that having a single formula to derive reputation leaves the system open to gaming, as rationally expected with almost any process that can be measured and quantified.ehavior . Key feaStack Exchange (stackexchange.com) is a collective intelligence system comprising multiple individual question and answer sites, many of which are already geared towards particular research communities, including maths and physics. The most popular site withinStack Exchange isStack Overflow, a community of software developers and a place where professionals exchange problems, ideas, and solutions.Stack Exchange works by having users publish a specific problem or question, and then others contribute to a discussion on that issue. This format is considered to be a form of dynamic publishing by some , a rapidly growing Bioinformatics resource, the use of which has contributed to the completion of traditional peer reviewed publications .PhysicsOverflow forms the counterpart forum toMathOverflow (https://mathoverflow.net/), with both containing a graduate-level question and answer forum, and an open problems section for collaboration on research issues. Both have a reviews section to complement formal journal-led peer review, where peers can submit preprints  for public peer evaluation, considered by most to be an “arXiv-2.0”. Responses are divided into reviews and comments, and given a score based on votes for originality and accuracy. Similar toReddit, there are moderators but these are democratically elected by the community itself. Motivation for engaging with these platforms comes from a personal desire to assist colleagues, progress research, and receive recognition for it  or negative. Karma provides a point-based reputation system for individuals, based not just on the quantity of engagement with the platform and its peers alone, but also on the quality and relevance of those engagements, as assessed by the wider engaging community (stackoverflow.com/help/whats-reputation). Peers have their status and moderation privileges within the platform upgraded as they gain reputation. Such automated privilege administration provides a strong social incentive for constructively engaging within the community. Furthermore, peers who asked the original questions mark answers considered to be the most correct, thereby acknowledging the most significant contributions while providing a stamp of trustworthiness. This has the additional consequence of reducing the strain of evaluation and information overload for other peers by facilitating more rapid decision making, a behavior based on simple cognitive heuristics   and are based at non-academic institutes. There is substantial scope in creating a reputation platform that goes beyond traditional measurements to include social network influence and open peer-to-peer engagement. Ultimately, this model can potentially transform the diversity of contributors to professional research and level the playing field for all types of formal contribution.setting .Stack OGit is a free and open-source distributed version control system developed by theLinux community in 2005 (git-scm.com/).GitHub, launched in 2008, works as a Web-basedGit service and has become the de facto social coding platform for collaborative and open source development and code sharing (github.com/). It holds many potentially desirable features that might be transferable to a system of peer review .GitHub also includes a variety of notification options for both users and project repositories. Users can watch repositories or files of interest and be notified of any new issues or commits (updates), and someone who has discussed an issue can also be notified of any new discussions of that same issue. Issues can also be tagged , and assigned to one or more participants, who are then responsible for that issue. Another item thatGitHub supports is a checklist, a set of items that have a binary state, which can be used to implement and store the status of a set of actions.GitHub also allows users to form organizations as a way of grouping contributors together to manage access to different repositories. All contributions are made public as a way for users to obtain merit.GitHub can be further measured quantitatively as a social product through the star-rating system, which is derived from the number of followers or watchers and the number of times a repository has been forked  or commented on. For scholarly research, this could ultimately shift the power dynamic in deciding what gets viewed and re-used away from editors, journals, or publishers to individual researchers. This then can potentially leverage a new mode of prestige, conferred through how work is engaged with and digested by the wider community and not by the packaging in which it is contained .Prestige atGitHub could be used to implement some style of peer evaluation and that it is well-suited to fine-grained iteration between reviewers, editors, and authors , created to give software developers a lightweight mechanism for software developers to quickly supplement their code with metadata and a descriptive paper, and then to submit this package for review and publication, andReScience (rescience.github.io), created to publish replication efforts in computational science.JOSS submission portal converts a submission into a newGitHub issue of type “pre-review” in theJOSS-review repository . The editor-in-chief checks a submission, and if deemed suitable for review, assigns it to a topic editor who in turn assigns it to one or more reviewers. The topic editor then issues a command that creates a new issue of type “review”, with a check-list of required elements for the review. Each reviewer performs their review by checking off elements of the review issue with which they are satisfied. When they feel the submitter needs to make changes to make an element of the submission acceptable, they can either add a new comment in the review issue, which the submitter will see immediately, or they can create a new issue in the repository where the submitted software and paper exist—which could also be onGitHub, but is not required to be—and reference said issue in the review. In either case, the submitter is automatically and immediately notified of the issue, prompting them to address the particular concern raised. This process can iterate repeatedly, as the goal ofJOSS is not to reject submissions but to work with submitters until their submissions are deemed acceptable. If there is a dispute, the topic editor  can weigh in. At the end of this process, when all items in the review check-list are resolved, the submission is accepted by the editor and the review issue is closed. However, it is still available and is linked from the accepted (and now published) submission. A good future option for this style of model could be to develop host-neutral standards using Git for peer review. For example, this could be applied by simply using a prescribed directory structure, such as:manuscript_version_1/peer_reviews, with open commenting via the issues function.TheJOSS usesGItHub’s issue mechanism,ReScience usesGItHub’s pull request mechanism: each submission is a pull request that is publicly reviewed and tested in order to guarantee that any researcher can re-use it. At least two reviewers evaluate and test the code and the accompanying material of a submission, continuously interacting with the authors through the pull request discussion section. If both reviewers can run the code and achieve the same results as were submitted by the author, the submission is accepted. If either reviewer fails to replicate the results before the deadline, the submission is rejected and authors are encouraged to resubmit an improved version later.WhileWikipedia is the freely available, multi-lingual, expandable encyclopedia of human knowledge (wikipedia.org/).Wikipedia, likeStack Exchange, is another collaborative authoring and review system whereby contributing communities are essentially unlimited in scope. It has become a strongly influential tool in both shaping the way science is performed and in improving equitable access to scientific information, due to the ease and level of provision of information that it provides. Under a constant and instantaneous process of reworking and updating, new articles in hundreds of languages are added on a daily basis.Wikipedia operates through a system of collective intelligence based on linking knowledge workers through social media , and therefore often for reasons of personal satisfaction. Edits occur as cumulative and iterative improvements, and due to such a collaborative model, explicitly defining page-authorship becomes a complex task. Moderation and quality control is provided by a community of experienced editors and software-facilitated removal of mistakes, which can also help to resolve conflicts caused by concurrent editing by multiple authors (wikipedia.org/wiki/Help:Edit_conflict). Platforms already exist that enable multiple authors to collaborate on a single document in real time, includingGoogle Docs,Overleaf, andAuthorea, which highlights the potential for this model to be extended into a wiki-style of peer review.PLOS Computational Biology is currently leading an experiment with Topic Pages (collections.plos.org/topic-pages), which are published papers subsequently added as a new page toWikipedia and then treated as a living document as they are enhanced by the community . It can be used for nominating potentially good articles that could become candidates for a featured article. Featured articles are considered to be the best articlesWikipedia has to offer, as determined by its editors and the fact that only ∼0.1% are selectively featured. Users submitting a new request are encouraged to review an article from those already listed, and encourage reviewers by replying promptly and appreciatively to comments. Compared to the conventional peer review process, where experts themselves participate in reviewing the work of another, the majority of the volunteers here, like most editors inWikipedia, lack formal expertise in the subject at hand . As such, although this is part of the process of conventional validation, such a system has little actual value onWikipedia due to its dynamic nature. Indeed, wiki-communities appear to have distinct values to academic communities, being based more on inclusive community participation and mediation than on trust, exclusivity, and identification (wikipedia.org/wiki/Wikipedia:Verifiability). Therefore, the process is perhaps best viewed as a process of “peer production”, but where attainment of the level of peer is relatively lower to that of an accredited expert. This provides a difference in community standing forWikipedia content, with value being conveyed through contemporariness, mediation of debate, and transparency of information, rather than any perception of authority as with traditional scholarly works .When applied to scholarly topics, this process clearly lacks the “peer” aspect of scholarly peer review, which can potentially lead to propagation of factual errors e.g.,. If seekication . Verifia3.5.2 Democratization of peer review. The advantage ofWikipedia over traditional review-then-publish processes comes from the fact that articles are enhanced consistently as new articles are integrated, statements are reworded, and factual errors are corrected as a form of iterative bootstrapping. Therefore, while one might consider aWikipedia page to be of insufficient quality relative to a peer reviewed article at a given moment in time, this does not preclude it from meeting that quality threshold in the future. Therefore,Wikipedia might be viewed as an information trade-off between accuracy and scale, but with a gap that is consistently being closed as the overall quality generally improves. Another major statement that aWikipedia-style of peer review makes is that rather than being exclusive, it is an inclusive process that anyone is allowed to participate in, and the barriers to entry are very low—anyone can potentially be granted peer status and participate in the debate and vetting of knowledge. This model of engagement also benefits from the “many eyes” hypothesis, where if something is visible to multiple people then, collectively, they are more likely to detect any errors in it, and tasks become more spread out as the size of a group increases. InWikipedia, and to a larger extentWikidata, automation or semi-automation through bots helps to maintain and update information on a large scale. For example,Wikidata is used as a centralized microbial genomics database , to one where the pool of potential peers is relatively large (wikipedia.org/wiki/Wikipedia:List_of_Wikipedians_by_number_of_edits). This is broadly similar to what is observed in current academic peer review systems, where the majority of the work is performed by a minority of the participants  that would need to be aligned with the norms and expectations of research communities. In the latter, author statements and peer reviews are considered valid because of the personal, identifiable status and reputation of authors, reviews and editors, which could be feasibly combined withWikipedia review models into a single solution. One example where this is beginning to happen already is with the WikiJournal User Group, which represents a publishing group of scholarly journals that apply academic peer review to their content . However, a more rigorous editorial review process is the reason why the original form ofWikipedia, known asNupedia, ultimately failed . There  failed . Future Hypothesis (web.hypothes.is) is a lightweight, portable Web annotation tool that operates across publishing platforms  to live alongside the content but also separate from it, allowing communities to form and spread across the internet and across content types, such as HTML, PDF, EPUB, or other formats  are used to federate or compile annotations for scholarly work. Reviewers often provide privately annotated versions of submitted manuscripts during conventional peer review, and Web annotation is part of the digitization of this process, while also decoupling it from journal hosts. A further benefit of Web annotations is that they are precise, since they can be applied in line rather than at the end of an article as is the case with formal commenting.Hypothesis and other Web annotation services is poorly recognized in scholarly communities, although such activities can be tied toORCID. However, there is definite value in services such asPubPeer, an online community mostly used for identifying cases of academic misconduct and fraud, perhaps best known for its user-led post-publication critique of aNature paper on STAP (Stimulus-Triggered Acquisition of Pluripotency) cells. This ultimately prompted the formal retraction of the paper, demonstrating that post-publication annotation and peer review, as a form of self-correction and fraud detection, can out-perform that of the conventional pre-publication process.PubPeer has also been leveraged as a way to mass-report post-publication checks for the soundness of statistical analyses. One large-scale analysis using a tool calledstatcheck . A blockchain connects a row of data blocks through a cryptographic function, with each block containing a time stamp and a link to the previous block in the chain. This system is decentralized, distributed, immutable, and transparent . Through a system of community trust, blockchains could be used to handle the following tasks:The blockchain technology could be leveraged to create a tokenized peer review system involving penalties for members who do no uphold the adopted standards and vice versa. A blockchain-powered peer-reviewed journal could be issued as a token system to reward contributors, reviewers, editors, commentators, forum participants, advisors, staff, consultants, and indirect service providers involved in scientific publishing . Such re1. Authenticating scientific papers (using time stamps and checksums), combating fraudulent science;2. Allowing and encouraging reviewers to actively engage in the scientific community;3. Rewarding reviewers for peer reviews with Science Coins;4. Allowing authors to contribute by giving Science Coins;5. Supporting verification and replicability of research.6. Keeping reviewers and authors anonymous, while providing a validated certification of their identity as researchers, and rewarding them.Ledger, launched in 2015, is the first academic journal that makes use of a system of digital signatures and time stamps based on blockchain technology . The aim is to generate irrevocable proof that a given manuscript existed on the date of publication. Another publishing platform being developed that leverages blockchain isAletheia, which uses the technology to “achieve a distributed and tamper proof database of information, storing document metadata, vote topics, vote results and information specific to users such as reputation and certifications” .This could help to improve the quality and responsiveness of peer reviews, as these are published publicly and the different participants are rewarded for their contributions. For instance, reviewers for a blockchain-powered peer-reviewed journal could invest tokens in their comments and get rewarded if the comment is upvoted by other reviewers and the authors. All tokens need to be spent in making comments or upvoting other comments. When the peer review is completed, reviewers get rewarded according to the quality of their remarks. In addition, the rewards can be attributed even if reviewer and author identity is kept secret; such a system can decouple the quality assessment of the reviews from the reviews themselves, such that reviewers get credited while their reviews are kept anonymous. Moreover, increased transparency and interaction is facilitated between authors, reviewers, the scientific community, and the public. The journalFurthermore, blockchain-based models offer the potential to go well beyond peer review, possibly integrating all functions of publication in general. They could be used to support data publication, research evaluation, incentivization, and research fund distribution. A relevant example is a proposed decentralized peer review group as a way of managing quality control in peer review via blockchain through a system of cohort-based training . This haAmazon now provides ML as a service through theirAmazon Web Services platform (aws.amazon.com/amazon-ai/),Google released their open source ML framework,TensorFlow (tensorflow.org/), andFacebook have similarly contributed code of theirTorch scientific learning framework (torch.ch/). ML has been very widely adopted in tackling various challenges, including image recognition, content recommendation, fraud detection, and energy optimization. In higher education, adoption has been limited to automated evaluation of teaching and assessment, and in particular for plagiarism detection. The primary benefits of Web-based peer assessment are limiting peer pressure, reducing management workload, increasing student collaboration and engagement, and improving the understanding of peers as to what critical assessment procedures involve , and neural network tools that may potentially assist with the peer review process. ML, as a technique, is rapidly becoming a service that can be utilized at a low cost by an increasing number of individuals. For example,statcheck . For example, does a clinical trial contain the appropriate registration information, are the appropriate consent statements in place, have new taxonomic names been registered, and does the research fit in with the existing body of published literature . The comatcheck . Under nThe second approach is to automatically determine the most appropriate reviewers for a submitted manuscript, by using a co-authorship network data structure . The advThirdly, given that machine-driven research has been used to generate substantial and significant novel results based on ML and neural networks, we should not be surprised if, in the future, they can have some form of predictive utility in the identification of novel results during peer review. In such a case, machine learning would be used to predict the future impact of a given work , and in effect to do the job of impact analysis and decision making instead of or alongside a human reviewer. We have to keep a close watch on this potential shift in practice as it comes with obvious potential pitfalls by encouraging even more editorial selectivity, especially when network analysis is involved. For example, research in which a low citation future is predicted would be more susceptible to rejection, irrespective of the inherent value of that research. Conversely, submissions with a high predicted citation impact would be given preferential treatment by editors and reviewers. Caution in any pre-publication judgements of research should therefore always be adopted, and not be used as a surrogate for assessing the real world impact of research through time. Machine learning is not about providing a total replacement for human input to peer review, but more how different tasks could be delegated or refined through automation.Scholastica (scholasticahq.com) includes real-time journal performance analytics that can be used to assess and improve the peer review process.Elsevier uses a system called Evise (elsevier.com/editors/evise) to check for plagiarism, recommend reviewers, and verify author profile information by linking toScopus. TheJournal of High Energy Physics uses automatic assignment to editors based on a keyword-driven algorithm , highlight the great promise of artificial intelligence in research and for application to peer review.Finally, it is important to note that ML and neural networks are largely considered to be conformist, so they have to be used with care , allows for detailed OPR of methods while also promoting reproducibility and refinement of techniques. This can help other scholars to begin work on related projects and test methodologies due to the openness of both the protocols themselves and the comments on them  of non-traditional products to receive academic credit, they must currently be integrated into the publication system that forms the basis for academic assessment and evaluation. Peer review of methodologies, such as media” .3.9.1 Software peer review. Software represents another area where traditional peer review has evolved. In software, peer review of code has been a standard part in computationally-intensive research for many years, particularly as a post-software creation check. Additionally, peer-programming  has been growing in popularity, especially as part of the Agile methodology, where it is employed as a check made during software creation . Academic code is sometimes released as open source, and many such released codebases have led to remarkable positive changes, with prominent examples including the Berkeley Software Distribution (BSD), upon which the Mac operating system (MacOS) is built; the ubiquitous TCP/IP Internet protocol stack; the Squid web proxy; the Xen hypervisor, which underpins many cloud computing infrastructures; Spark, the big data stream processing framework; and the Weka machine learning suite.When it comes to software development, there is a dichotomy of review practices. On one hand, software developed in open source communities  relies on peer review as an intrinsic part of its existence, from creation and through continual evolution. On the other hand, software created in academia is typically not subjected to the same level of scrutiny. For the most part, at present there is no requirement for software used to analyze and present data in scholarly publications to be released as part of the publication process, let alone be closely checked as part of the review process, though this may be changing due to government mandates and community concerns about reproducibility. One example from Computer Science is ACM SIGPLAN’s Programming Language Design and Implementation conference that encourages the submission of supporting material (including code) for review by a separate technical committee. Papers with successfully evaluated artifacts get stamped with seals of approval visible in the conference proceedings. ACM is adopting a similar strategy on a wider scale through its Task Force on Data, Software, and Reproducibility in Publication . At first, peer review for these software articles was the same as for any other paper, but this is changing now, particularly as journals specializing in software , theJournal of Open Research Software , theJournal of Open Source Software ) are emerging. The material that is reviewed for these journals is both the text and the software. ForSoftwareX  andJORS (openresearchsoftware. metajnl.com/about/#q4), the text and the software are reviewed equally. ForJOSS, the review process is more focused on the software (based on therOpenSci model (joss.theoj.org/about#reviewer_guidelines). The purpose of the review also varies across these journals. InSoftwareX andJORS, the goal of the review is to decide if the paper is acceptable and to improve it through a non-public editor-mediated iteration with the authors and the anonymous reviewers. While inJOSS, the goal is to accept most papers after improving them if needed, with the reviewers and authors ideally communicating directly and publicly throughGitHub issues. Although submitting source code is still not required for most peer review processes, attitudes are slowly changing. As such, authors increasingly publish works presented at major conferences (which are the main channel of dissemination in computer science) as open source, and also increasingly adopting the use ofarXiv as a publication venue . The next version of the document was placed on a website, and new reviewers commented on it both through additionalGitHub issues and throughHypothesis (via.hypothes.is/https://www.force11.org/software-citation-principles [see Section 3.6]). Finally, the document was submitted toPeerJ Computer Science, which used a pre-publication review process that allowed reviewers to sign their reviews and the reviews to be made public along with the paper authors’ responses after the final paper was accepted and published , or traditional data management plans for review between text and data. Using such hybrid evaluation methods could prove to be quite successful, not just for reforming the peer review process, but also to improve the quality and impact of scientific publications. One could envision such a hybrid system with elements from the different models we have discussed.While individual publishers may use specific methods when peer review is controlled by the author of the document to be reviewed, multiple peer review models can be used either in series or in parallel. For example, theblished . The autquality control,certification, andincentivization. Only by harmonizing all three of these, while grounding development in diverse community stakeholder engagement, can the implementation of any future model of peer review be ultimately successful. Such a system has the potential to greatly disrupt the current coupling between peer review and journals, and lead to an overhaul of digital scholarly communication to become one that is fit for the modern research environment.In Section 3, we summarized a range of social and technological traits of a range of individual existing social platforms. Each of these can, in theory, be applied to address specific social or technical criticisms of conventional peer review, as outlined in Section 2. Many of them are overlapping and can be modeled into, and leveraged for, a single hybrid platform. The advantage is that they each relate to the core non-independent features required for any modern peer review process or platform:Wikipedia andReddit, it is clear that community self-organization and governance represent a possible alternative when combined with a core community of moderators. These moderators would have the same operational functionality as editors in terms of gate-keeping and facilitating the process of engagement, but combined with the role of a Web forum moderator. Research communities could elect groups of moderators based on expertise, prior engagement with peer review, and transparent assessment of their reputation. This layer of moderation could be fully transparent in terms of identity by using persistent identifiers such asORCID. The role of such moderators could be essentially identical to that of journal editors, in soliciting reviews from experts, making sure there is an even spread of review attention, and mediating discussions. Different communities could have different norms and procedures to govern content and engagement, and to self-organize into individual but connected platforms, similar toStack Exchange orReddit.ORCID has a further potential role of providing the possibility for a public archive of researcher information and metadata  that can be leveraged using automated techniques to match potential referees to items of interest, while avoiding conflicts of interest.Quality control is often hailed as the core function of peer review, but is invariably difficult to measure. Typically, it has been administered in a closed system, where editorial management formed the basis. A strong coupling of peer review to journals plays an important part in this, due to the association of researcher prestige with journal brand as a proxy for quality. By looking at platforms such asGitHub, quality control is provided through a system of automated but managed invited review, public interaction and collaboration (like withStack Exchange), and transparent refinement. This would also help prevent a situation where “the rich get richer”, as semi-automation ensures that all content has the same chance of being interacted with. Engagement could be conducted via a system of issues and public comments, as onGitHub, where the process is not to reject submissions, but to provide a system of constant improvement. Such a system is already implemented successfully atJOSS. Both community moderation and crowd sourcing would play an important role here to prevent underdeveloped feedback that is not constructive and could delay efficient manuscript progress. This could be further integrated with a blockchain process so that each addition to the process is transparent and verifiable.In such a system, published objects could be preprints, data, code, or any other digital research output. If these are combined with management through version control, similar toGitHub issues), indexed, and the latest version is assigned a persistent identifier, such as aCrossRef DOI, as well as an appropriate license. If desired, these objects could then form the basis for submissions to journals, perhaps even fast-tracking them as the communication and quality control would already have been completed. Such a process would promote inclusive participation, community interaction, and quality would become a transparent function of how information is engaged with, digested, and reused. The role of peer review would then be coupled with the concept of a “living published unit”, independent of journals themselves. The role of journals and publishers would be dependent on how well they justify their added value, once community-wide and public dissemination and peer review have been decoupled from them.When authors and moderators deem the review process to have been sufficient for an object to have reached a community-decided level of quality or acceptance, threads can be closed , combined with their reputation as authors. Rather than having anonymous or pseudonymous participants, for peer review to work well, it would require full identification, to connect on-platform reputation and authorship history. Rather than a journal-based form, certification is granted based on continuing engagement with the research process and is revealed at the article (or object) and individual level. Communities would need to decide whether or not to set engagement filters based on quantitative measures of experience or reputation, and what this should be for different activities. This should be highly appealing not just to researchers, but also to those in charge of hiring, tenure, promotion, grant funding, ethical review and research assessment, and therefore could become an important factor in future policy development. Models likeStack Exchange are ideal candidates for such a system, because achievement of certification takes place via a process of community engagement and can be quantified through a simple and transparent up-voting and down-voting scheme, combined with achievement badges. Any outputs from assessment could be portable and applied toORCID profiles, external webpages, and continuously updated and refined through further activity. While a star system does not seem appealing due to the inherent biases associated with it, this quantitative way of “reviewing the reviewers” creates a form of dynamic social reputation. As this is decoupled from journals, it alleviates all of the well-known issues with journal-based ranking systems  is perhaps unwise, as it might lead to a distortion of incentives.de facto standards that help to build towards consistent practice and adoption.None of the ideas we have proposed here are particularly radical, representing more the recombination of existing variants that have succeeded or failed to varying degrees. We have presented them here in the context of historical developments and current criticisms of peer review in the hope that they inspire further discussion and innovation. A key challenge that our proposed hypothetical hybrid system will have to overcome is simultaneous uptake across the whole scholarly ecosystem. This in turn will most likely require substantial evidence that such an alternative system is more effective than the traditional processes ;A new journal,2. peerreviewcongress.org/index.html;The International Congress on Peer Review and Scientific Publication, which aims to encourage research into the quality and credibility of peer review (3. peere.org/).The PEERE initiative, which has the objectives of improving the efficiency, transparency and accountability of peer review (While we briefly considered peer review in the context of some non-text products here (Section 3.9), there is clear scope for further discussion of the diverse applications of peer review. The utility of peer review for research grant proposals would be a fruitful avenue for future work, given that here it is less about providing feedback for authors, and more about making assessments of research quality. There are different challenges and different potential solutions to consider, but with some parallels to that discussed in the present manuscript. For example, how does the role of peer review for grants change for different applicant demographics in a time when funding budgets are, in some cases, being decreased, but in concert with increasing demand and research outputs.Slack orGitter. These are widely used for project communication and operate analogous to a real-time collaboration system with instantaneous and continuous “peer review”. While such activities can be used to supplement other hybrid platforms, as an independent or stand-alone mode of peer review, the concept is quite distant from the other models that have been discussed here .One further aspect that we did not examine in detail is the use of instant messaging services, likeIf the current system of peer review were to undergo peer review, it would undoubtedly achieve a “revise and resubmit” decision. AsDataCite,CrossRef, andORCID will be essential in binding the social and technical aspects of this to an interoperable, sustainable and open scholarly infrastructure  to create an optimal system of peer review aligned with the diverse needs of non-independent research communities. By decoupling peer review, and with it scholarly communication, from commercial entities and journals, it is possible to return it to the core principles upon which it was founded as a community-based process. Through this, knowledge generation and access can become a more democratic process, and academics can fulfil the criteria that have been entrusted to them as creators and guardians of knowledge. General comments On reading this again, I still think it is very long and overly repetitive (for example the various types of open peer review are discussed in a number of sections) and the paper would benefit from substantial consolidation of key topics. I recognize, however, that the authors are unlikely to be able to substantially change it now, given the large and diverse authorship. I note that a number of changes were made, though as far I can see the authors didn’t respond specifically to all the points I made, hence a number of my previous comments still stand. I'll leave it up to the authors if they respond to these comments. 1.0.1 Methods Thanks for clarifying that there was not a systematic review process here. I think it would be helped if the limitation of the approach taken was noted within the methodology . It is clearer that the paper does express a multitude of perspectives, rather than being definitive. The authors argue against considering peer review as a “gold standard” but again on re-reading it I think the overall tone of the article may actually risk perpetuating the myth that peer review is some sort of panacea, rather than essentially a QC process. Anything that helps in demystification of the process  would be helpful in encouraging a healthy debate in this area.“Now, the research community has the opportunity to help create an efficient and socially-responsible system of peer review.”, though I think it is more likely there will be a multitude of systems that we should be looking to support. I agree with this statement in the conclusionsSpecific comments There are a number of typos and sentences that need clarification - I have noted those I found. Italics are used when quoting from the paper IntroductionHowever, peer review is still often perceived as a “gold standard” ;Mayden (2012)), despite the inherent diversity of the process and never intended to be used as such. There seems to be text missing from this sentenceBy allowing the process of peer review to become managed by a hyper-competitive industry, developments in scholarly publishing have become strongly coupled to the transforming nature of academic research institutes.  I am not sure what this sentence means. 1.3equator-network.org) have been important to improve the reporting of research and its peer review according to standardised criteria. Another response has been COPE, the Committee on Publication Ethics , was established in 1997 to address potential cases of abuse and misconduct during the publication process. , and later created specific guidelines for peer review. I don’t think that EQUATOR was set up as a result of specific criticisms about peer review – it was set up to address poor reporting of papers, and COPE certainly wasn’t set up to address peer review. COPE did later produce guidelines and other resources for peer reviewers, which have been recently updated – see https://publicationethics.org/peerreviewA popular editorial in The BMJ stated that made some quiter serious (typo “quiter”) 2.2.1 Most statistical reviewers are paid. This is not mentioned in the textThe Committee on Publication Ethics (COPE) could be used as the basis for developing formal mechanisms adapted to innovative models of peer review, including those outlined in this paper.   2.4.2 COPE does advise on new peer review models as appropriate to ethics cases so I am not sure what is meant here. 2.5.1Papers on digital repositories are cited on a daily basis and much research builds upon them, although they may suffer from a stigma of not having the scientific stamp of approval of peer review Should specify that this section is for preprint repositories 2.5.2 It’s a shame that clinical trial registration is not discussed more, though I appreciate it being added. It’s an interesting example of how innovations in one specialty often are not fully appreciated in another .I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. The revisions are good. The paper is now more mature. I am comfortable accepting it for indexing. I noted a typo - 2.4.2 The Rodriguez-Bravo et al should have a "In" prior to "Press".I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Thank you for asking me to review this paper. Ironically, but perhaps not surprisingly, this was quite a hard paper to peer review and I don’t claim that this peer review does anything more than provide one (non-exhaustive) opinion on this paper. My views on peer review, which have formed over more than 15 years of being involved in editing and managing peer review will have coloured my peer review here. I think it's useful to regard all journal processes, which includes peer review, as components of the QC which begins with checking for basics such as the presence of ethics statements or trial registration, or the use of  reporting guidelines for example, through to in depth methodological review. I don't think that any of the parts of the system of QC, including peer review, are perfect but the system is one component of attempting to ensure reproducibility, itself a core role of journals. The very basic functions of QC  are often not given enough emphasis, though they are going to become more important as, for example, other types of publication such as  preprints increase in popularity.General Comments This is a wide ranging, timely paper and will be a useful resource. My main comment is that this is a mix of opinion, review, and thought experiment of future models. While all of these are needed in this area, for the review part of the paper, it would be much strengthened with a description of the methodology used for the review, including databases searched for information and keywords used to search, etc. The paper is very long and there is a substantial amount of repetition. I think the introduction in particular could be much shortened - especially as it contains a lot of opinion, and repetition of issues dealt with elsewhere in the paper.Non-transparent,impossible to evaluate,biased, secretive, exclusive The language of the paper is also quite emotive in places and though I would personally agree with some of the sentiments I don't think they are helpful in making the authors’ case eg in Table 2  assessment of pre publication peer review is listed asThe entrenchment of the ubiquitously practiced and much more favored traditional model  is ironically non-traditional, but nonetheless currently revered.   Or I think it worth reviewing the language of the paper with that in mind. Although it arises in a number of places I don’t feel the authors address fully the complexity of interdisciplinary differences. The introduction would have been a good place to set this down. There is no mention of initiatives such as EQUATOR which have been important in improving reporting of research and its peer review. http://www.equator-network.org/ I was surprised to see very little discussion of the problems associated with commenting - especially of tone - that can arise on anonymous or pseudonymous sites such as Pubpeer and reddit.1. There was no discussion of post publication reviews which originate in debates on twitter. There have been some notable examples of substantial peer review happening - or at least beginning there eg that on arsenic lifeSelf Journal of Science. There are quite a few places where initiatives are mentioned but not referenced or hyperlinked. egSpecific comments Introductiongold standard”. In my view many of the issues arising from peer review are that it is held to a standard that was never intended for it. I would take issue with the term “ Introduction paragraph 2 - where PLOS is mentioned here it should be replaced by PLOS ONE - the other journals from PLOS have other criteria for review. I am surprised that PLOS ONE does not get more of a mention in how much of a shift it represent in its model of uncoupling objective from subjective peer review, and how it led to the entire model for mega journals.“The purpose of developing peer reviewed journals became part of a process to deliver research to both generalist and specialist audiences, and improve the status of societies and fulfil their scholarly missions”  1.1.1 I think it is worth noting that another function of peer review at journals was that it was part of earliest attempts of ensuring reproducibility - which is of course a very hot topic nowadays but in fact has its roots right back to when experiments were first described in journals.“From these early developments, the process of independent review of scientific reports by acknowledged experts gradually emerged. However, the review process was more similar to non-scholarly publishing, as the editors were the only ones to appraise manuscripts before printing”   There is a misconception here, which I think is quite common. In the vast majority of cases editors are also peers, and may well be “acknowledged experts” - in fact certainly will be at society journals. The distinction between editors and peer reviews can be a false one with regard to expertise.1.1.2 where publishers call upon external specialists to validate journal submissions.  It is important to note that it is editors who manage review processes. Publisher are largely responsible for the business processes; editors for the editorial processes.By allowing the process of peer review to become managed by a hyper-competitive industry, developments in scholarly publishing have become strongly coupled to the transforming nature of academic research institutes.  “ These have evolved into internationally competitive businesses that strive for quality through publisher-mediated journals by attempting to align these products with the academic ideal of research excellence ” I am not sure what is meant by “these” in this second sentence, nor what is meant by a “publisher-mediated  journal”. Virtually all journals have a publisher - even small academic-led ones.1.1.3  This practice represents a significant shift, as public dissemination was decoupled from a traditional peer review process, resulting in increased visibility and citation rates .  Many papers posted on arxiv.org do go on to be published in peer reviewed journals. Are these references referring to  increased citation of the preprints or the version published in a peer reviewed journal?The launch of Open Journal Systems  in 2001 offered a step towards bringing journals and peer review back to their community-led roots.   The jump here is odd. OJS actually can support a number of models of peer review, including a traditional model of peer review, just on a low cost open source platform, not a commercial one. The innovation here is the technology.Digital-born journals, such as PLOS ONE, introduced commenting on published papers.  http://www.bmj.com/rapid-responses. Here the reference should be to all of PLOS as commenting was not unique to PLOS ONE. However, the better example of commenting is the BMJ which had a vibrant paper letters page which it transformed very successfully to its rapid responses - and it remains the  journal that has had most success  Other services, such as Publons, enable reviewers to claim recognition for their activities as referees. Originally Academic Karma http://academickarma.org/ had a similar purpose though now it has a different model - facilitating peer review of preprints. Figure 2 PLOS ONE and ELife should be added to this timeline. Elife’s collaborative peer review model is very innovative. I am not sure why Wikipedia is in here.1.3 One consequence of this is that COPE, the Committee on Publication Ethics (publicationethics.org), was established in 1997 to address potential cases of abuse and misconduct during the publication process. COPE was first established because of issues related to author misconduct which had been identified by editors. Though it does now have a number of cases relating to peer review , the guidelines for peer review came much later and peer review was not an early focus.Taken together, this should be extremely worrisome, especially given that traditional peer review is still viewed almost dogmatically as a gold standard for the publication of research results, and as the process which mediates knowledge dissemination to the public. Just as a washing machine has a quality kite-mark, peer review is a kind of quality mark for science. It tells you that the research has been conducted and presented to a standard that other scientists accept. At the same time, it is not saying that the research is perfect (nor that a washing machine will never break down). http://senseaboutscience.org/wp-content/uploads/2016/09/peer-review-the-nuts-and-bolts.pdf I am not sure I would agree. Every person I know who works in publishing accepts that peer review is an imperfect system and that there is room for rethinking the process. Sense about Science puts it well in its guide: ” Table 2. Note that quite a few of these approaches can co-exist. Under post publication commenting PLOS ONE should be PLOS. BMJ should be added here. 1.4 Quite a lot of subscription journals do reward reviewers by providing free subscriptions to the journal - or OA journals provide discounts on APCs (including F1000). Furthermore, some reviewers are paid, especially statistical reviewers.Hence, this [Wiley] survey could represent a biased view of the actual situation.   2.2.2    I’d like to see evidence to support this statement.2.2.3  The idea here is that by being able to standardize peer review activities, it becomes easier to describe, attribute, and therefore recognize and reward them  I think the idea is to standardise the description of peer review, not the activity itself. Please clarify.Either way, there is little documented evidence that such retaliations actually occur either commonly or systematically. If they did, then publishers that employ this model such as Frontiers or BioMed Central would be under serious question, instead of thriving as they are.   2.4.2. This sentence seems to be in contradiction to the phrase below:In an ideal world, we would expect that strong, honest, and constructive feedback is well received by authors, no matter their career stage. Yet, it seems that this is not the case, or at least there seems to be the very real perception that it is not, and this is just as important from a social perspective. Retaliations to referees in such a negative manner represent serious cases of academic misconduct2.5.1. This process is mediated by ORCID for quality control, and CrossRef and Creative Commons licensing for appropriate recognition. They are essentially equivalent to community-mediated overlay journals, but with the difference that they also draw on additional sources beyond pre-prints.   This is an odd description. In what way does ORCID mediate for quality control?2.5.2 Two-stage peer review and Registered Reports.  Registration of clinical trials predated registered reports by a number of years and it would be useful to include clinical trial registration in this section.3 Potential future models   NB I didn’t review this section in detail.3.5 as was originally the case with Open Access publishing,   The perception of low quality in OA was artificially perpetuated by traditional publishers more than anything else - it was not inherent to the process.2. 3.5 Wikipedia and PLOS Computational Biology collaborated in a novel peer review experiment which would be worth mentioning  - see http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002446 3.9.2 Data peer review. This is  a vast topic and there are many initiatives in this area, which are not really discussed at all. I would suggest this section should come out - especially as earlier on it is noted that the paper focuses mainly on peer review of traditional papers. I would also suggest taking out the parts on OER and books.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. This manuscript is a herculean effort and enjoyable read. I learned lots, (and I think the paper will be a good resource for anybody interested in the field of peer review) which for me is usually a good sign of a paper’s worth. The authors report on many aspects of peer review and devote considerable attention to some challenges in the field and the enormous innovation the field is witnessing. I think the paper can be improved: 1. It is missing a Methods section. It was unclear to me whether the authors conducted a systematic review or whether they used a snowballing technique (starting with seed articles) to identify the content discussed in the paper? Did the authors search electronic databases (and if so which ones?) What were their search strategies and/or did they rely on their own file drawers? Are all the peer review innovations/systems/approaches identified by the authors discussed or did they only discuss some ? With a focus on reproducibility I think the authors need to document their methods.1 in their conclusions. I think the evidence argument needs to be made more strongly in the body of the paper). I do not think the paper is strong enough regarding the large swaths of the peer review processes (current and innovations) for which there is no evidence2 and it is difficult to gain access to peer reviews to better understand their processes and effectiveness – open the black box of peer review3. 2. I think the authors missed an important opportunity to discuss more deeply the need for evidence with all the current and emerging peer review systems . 3. There is limited data to inform us about several of the current peer review systems and innovations. In clinical medicine new drugs do not simply enter the market. They need undergo a rigorous series of evaluations, typically randomized trials prior to approval. Shouldn’t we expect something similar for peer review in the marketplace? It seems to me that any peer review process/innovation in development or released should have an evaluation  component integrated into it. Without evaluation we will miss the opportunity to generate data as to the effectiveness of the different peer review systems and processes. Research is central to getting a better understanding of peer review. It might useful for the authors to mention the existence of some groups/outlets committed to such research – PEERE (http://www.peere.org/) and the International Congress on Peer Review and Scientific Publication (http://www.peerreviewcongress.org/index.html). There is also a new journal committed to publishing peer review research  Jefferson 2002 Concerning measurement of peer review, there is accumulating evidence that there is little agreement as to how best to measure it. Unlike clinical medicine where there is a growing recognition of the need for core outcome set assessments (http://www.comet-initiative.org/) across all studies within specific content areas  we have not yet developed such an approach for peer review. Without a core outcome set for measuring peer review it will continue to be difficult to know what components of peer review researchers are trying to measure. Similarly, without a core outcome set it will be difficult to aggregate estimates of peer review across studies . Concerning the second point – there is little agreement as to an optimal design to evaluate the effectiveness of peer review. This is a critical issue to remedy in any effort to assess the effectiveness of peer review. 5. The paper assumes (at least that’s how I’ve interpreted it – the paper is silent on this issue) that peer reviewers are all similarly proficient in peer reviewing. There is little training for peer reviewers (new efforts by some organizations such as Publons Academy are trying to remedy this). I started my peer-reviewing career without any training, as did many of my colleagues. If we do not train peer reviewers to a minimum globally accepted standard we will fail to make peer review better.  6. 6. Peer review does not function in a vacuum. The larger ecosystem includes other players’ most notably scientific editors. There is little discussion in the paper about this relationship and its potential (dys)function 7. In section 2.2.1 you could also add at least one journal has an annual prize for peer reviewing  8. In the competing interests section of the paper it indicates that the first author works at ScienceOpen although the affiliation given in the paper is Imperial College London. Is this a joint appointment? Clarification is needed. A similar clarification is required for TRH.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.