Journal peer review regulates the flow of ideas through an academic discipline and thus has the power to shape what a research community knows, actively investigates, and recommends to policymakers and the wider public. We might assume that editors can identify the ‘best’ experts and rely on them for peer review. But decades of research on both expert decision-making and peer review suggests they cannot. In the absence of a clear criterion for demarcating reliable, insightful, and accurate expert assessors of research quality, the best safeguard against unwanted biases and uneven power distributions is to introduce greater transparency and structure into the process. This paper argues that peer review would therefore benefit from applying a series of evidence-based recommendations from the empirical literature on structured expert elicitation. We highlight individual and group characteristics that contribute to higher quality judgements, and elements of elicitation protocols that reduce bias, promote constructive discussion, and enable opinions to be objectively and transparently aggregated. Trust in the good judgement of reviewers with relevant qualifications, experience, and scientific skill is at the heart of peer review. Editors value reviewers’ expertise, and multiple studies have found a strong correlation between editorial final decisions and reviewers’ judgements –3. HowevContemporary peer review is predominantly a journal-organised, pre-publication quality assurance activity wherein independent and (often) anonymous reviewers provide their opinion on the suitability of submissions for publication , 13, 14,Our recommendations are based on our collective experience developing and implementing the IDEA protocol  reviews . AccessiWe tend to credit people with expertise based on their status, publications, affiliation with well-known institutions etc., yet such credentials have been shown to be unreliable criteria. Studies have shown mixed results for the association between traditional markers of expertise and indicators of judgement performance, such as overconfidence , reliabiSeveral large-scale projects have shown peer-reviewed research in medicine, psychology and economics often fails to replicate –63, raisOne aim of structuring elicitation processes is to quantify (but not eliminate) uncertainty. Structured expert elicitation protocols achieve this by standardising the response formats of quantitative elicitation questions, and IDEA, for instance, asks experts to provide bounds that give a measure of uncertainty. The procedure removes consensus pressure and associated biases by aggregating individual judgements mathematically rather than behaviourally. In peer review, this can be achieved by structuring judgements around a predefined set of questions about research quality, expressed in quantitative terms .Importantly, many structured expert elicitation protocols complement quantitative estimation in several ways, i.e. by collecting written feedback, facilitating discussion among experts, encouraging experts to correct misunderstandings, and by exploring diverse perspectives and counterfactual evidence without forcing consensus. A peer review process modelled after the IDEA protocol will generate both quantitative and qualitative data that will feed into editorial decision-making.Nevertheless, the use of numerical scores may give an unwarranted impression of precision. Indeed, the extent to which different reviewers and/or the editors share conceptualisations or reference points for these scores is questionable and peer reviewers often are miscalibrated –70. NotwWhen faced with uncertainty, individuals – including experts – use heuristics, or cognitive shortcuts, that may lead to erroneous conclusions . Group iApplied to peer review, the interactive process we envision encourages reviewers to articulate their reasoning explicitly before sharing it with others, and subsequently to probe the rationales that underpin alternative judgments. It also promotes the resolution of conflicts due to misunderstandings, lack of relevant knowledge, or the application of idiosyncratic evaluation criteria , 87–90. While interactions among reviewers are still relatively uncommon in journal peer review, interviews with editors and reviewers uncovered a practice of conferring with colleagues on how to review a manuscript and collective decision-making . This suTo further mitigate potential pernicious social influences that can undermine the wisdom of crowd , it is iAnonymity is a contested feature of traditional peer review protocols. Critics claim that it encourages laziness, idiosyncratic (or self-interested) reviews, responsibility and even abuses of power , 92. OpeEfforts are underway to support the implementation of more transparent peer-reviewing practices –95. This


Although established forms of peer review are often criticized for being slow, secretive, and even unfair, they are repeatedly mentioned by academics as the most important indicator of quality in scholarly publishing. In many countries, the peer review of books is a less codified practice than that of journal articles or conference papers, and the processes and actors involved are far from uniform. In Sweden, the review process of books has seldom been formalized. However, more formal peer review of books has been identified as a response to the increasing importance placed on streamlined peer-reviewed publishing of journal articles in English, which has been described as a direct challenge to more pluralistic publication patterns found particularly in the humanities. In this study, we focus on a novel approach to book review, Kriterium, where an independent portal maintained by academic institutions oversees the reviewing of academic books. The portal administers peer reviews, providing a mark of quality through a process which involves reviewers, an academic coordinator, and an editorial board. The paper studies how this process functions in practice by exploring materials concerning 24 scholarly books reviewed within Kriterium. Our analysis specifically targets tensions identified in the process of reviewing books with a focus on three main themes, namely the intended audience, the edited volume, and the novel role of the academic coordinator. Moreover, we find that the two main aims of the portal–quality enhancement (making research better) and certification –are recurrent in deliberations made in the peer review process. Consequently, we argue that reviewing procedures and criteria of quality are negotiated within a broader discussion where more traditional forms of publishing are challenged by new standards and evaluation practices. Peer assessment, in more formalized peer review procedures and in less codified contexts such as seminars, is the main evaluation procedure in academia. Peer review comes in many forms and may involve elaborate systems of double-blind review or more loosely organized assessment by editors or colleagues . The groThe stress on peer review in the assessment of research and the continuing importance of book publishing in many fields have highlighted the need for the development and formalization of peer review of books. For instance, initiatives in Flanders have led to the creation of a “label for peer-reviewed books” , where tIn this paper, we study Kriterium, an initiative that aims to facilitate peer review for books in a Swedish context. Kriterium was launched in the autumn of 2015, and the first six books were published in 2016. It functions as a national portal for administering peer review for academic books by various publishers, including Swedish publishing houses and university publication series. The review process is managed by the Kriterium board, which primarily consists of established researchers, and administered by an academic coordinator who is specially appointed for each manuscript. Inclusion in Kriterium’s series also requires that the accepted book is parallel published open access on Kriterium’s website. As documented by Based on an analysis of documentation created as part of Kriterium’s peer review process, the aim of the paper is to better understand how the review process and its structure serves to achieve the twin goals of quality enhancement and quality certification of scholarly works. We have paid particular attention to whether there are certain challenges, themes or types of recommendations that are recurring, and that may be specific to the review of scholarly books and to the Kriterium platform. This has led us to focus on how the intended audience is discussed by reviewers; how edited volumes are reviewed; and how the role of the academic coordinator is shaped in relation to authors and volume editors. Following a general introduction to the review of books and to Kriterium, these issues will each be developed in separate sections. Furthermore, with an eye towards the contested role of the scholarly book that prompted the development of Kriterium in the first place, we have looked for instances in the review process where inherent tensions existing in the social sciences and the humanities might become visible.Although often treated as a uniform process, academic publishing practices differ greatly depending on genre, country, and discipline, albeit the increasingly international academic system leads to more and more standardization. National differences in tradition are apparent not least when it comes to the quality control of books, a topic that has been researched considerably less than that of journal articles . In the In several European countries, including Sweden, monograph publishing has traditionally not been preceded by blind peer review; rather, scholarly quality is confirmed in other ways . In someCountries with small languages, and consequently a small potential audience for academic or professional books, often rely on print subsidies for monograph publishing . SubsidiAlthough the goal of Kriterium primarily was to raise the status of the monograph by ensuring that they could be peer reviewed, edited volumes have made up an almost equal share of manuscripts processed. Peer review of the edited volume format, however, presents its own challenges, some of which could presumably bring the inherent tension between traditional publishing ideals in the humanities and the quality control associated with peer review to the fore . While tCompared to journal articles, Giménez-Toledo et al.  note that assessment criteria for books are “fuzzy and unclear.” Peer review of edited volumes seldom, if ever, entail the same level of scrutiny as journal articles (or even monographs) and the practice is far from standardized. As a result, there is great variance between publishing houses (or editors) and individual reviewers. Some peer review processes simply look at the volume as a whole, whereas others might provide more in-depth quality control of individual chapters. There are ostensibly several factors that add to the difficulty in adopting peer review formats for individual journal articles to edited volumes. These include the fact that the latter must deal with multiple authors, often from diverse scholarly backgrounds and traditions brandishing different ideals on, for instance, style, methodology, and theoretical perspectives. Finding reviewers that can cover such a wide range is difficult and the issue of where to place the bar for scholarly rigor might prove challenging. Moreover, whereas articles are commonly written with peer review in mind, edited volumes can often be the result of either workshops and conferences or impromptu discussions within informal networks and between local colleagues, situations that do not necessarily take the publishing end of the process into consideration 2. ThisToday, Kriterium is described as a “portal for review, publication and dissemination of high-quality academic books, in accordance with the principles for open access.”Concerns were raised at this early stage over new publication standards and metrics, including the introduction of “publication points” that might threaten the traditional modes of scholarly publication in the social sciences and humanities. Early ambitions targeted the so called “Norwegian list”, a register of publication outlets whose importance has grown in Sweden over the past decade, and which is used by some academic institutions to evaluate the performance of institutions and faculties . When KrAnother related motivating factor was the fear that Swedish as an academic language would further diminish if scholars were rewarded for publishing articles in English. This idea coalesced with an ambition to protect the existence of Swedish publishing houses (and the academic series published by universities) aimed at scholarly monographs, and to combat the commercialization of academic publishing . The SweFormally, Kriterium can be described as an intermediary, an infrastructure that provides peer review for authors and publishers who wish their books to have a stamp of qualityPeer review procedures and editorial work processes are often secretive, and it may be difficult to access material documenting the process. In this case, Kriterium has generously shared the full documentation of board meetings, including peer review reports, as well as communication between the academic coordinator, the authors, and the editorial board. This is a rich material, consisting of hundreds of items, and for the purpose of this paper we have focused on a selection of the available documents. The focus has been on the process of peer review. Two types of documents, namely reports from reviewers and from academic coordinators, have been singled out for further analysis. It should be noted that many reviewers also make suggestions for changes directly in the manuscript. However, these more editorial types of comments, although important, have not been included in the analysis. Overall, the material consists of the reports available for 24 books: 14 edited volumes and 10 monographs. In a few cases, one of the reports was missing. Seven edited volumes were written in English while the remaining 16 books, including all monographs, were in Swedish. Kriterium mainly targets the social sciences and humanities and the manuscripts analyzed come from a wide range of fields including history, archeology, literature, Romance languages, art history, political science, ethnology, and the history of ideas, and they are often interdisciplinary.A total of 49 review reports, and 28 recommendations by academic coordinators were studied. For the purpose of pseudonymization, we will refer to the books as monograph a, b, c  and edited volume a, b, c , and as review report  or coordinator report . A few manuscripts are scholarly editions and collections of letters; genres we considered too unique to include in the study. Although the material we have had access to is not complete, we have deemed that it is substantial enough to provide an insight into the review process at Kriterium. Of the 24 analyzed books, 20 had been published in September 2021, a couple more are accepted for publication, and a few have been rejected or withdrawn.The secretive, and potentially sensitive, nature of the material makes it important to assert that these documents have been handled and analyzed with confidentiality. Given that the social sciences and humanities in Sweden comprise a comparably small context and that the number of books published is limited, we refrain from including information that may identify a specific book when we present quotes or in other ways report our results. In some cases, the quotes have also been translated into English. The people involved in the process may, of course, be able to identify quotes from documents they have had access to.The material was analyzed through qualitative content analysis . In the Before moving on to the identified themes, we first present and discuss the context in the form of the review process and work routines at Kriterium. This section is based on texts about Kriterium as well as on the analysis of review reports.The first step in the process of having a book provided with the Kriterium label involves the submission of a book proposal from a publisher or university publication series to the board of Kriterium. The proposal includes a brief description of the book as well as the full manuscript or at least one chapter. If the proposal is deemed to be a suitable contribution , the board will appoint an academic coordinator. The book publisher is asked to suggest a suitable coordinator in their proposal. This should be an established researcher within the field of the manuscript, and a person who has not previously collaborated or are colleagues with the author(s). If the book is suggested by a university publication series, the series’ editor will often act as academic coordinator, while the role will be filled by an “ad-hoc” coordinator if the book is to be published by a publishing house. The academic coordinator will suggest two reviewers who will read the full manuscript, be a link between author(s)/editor(s) and reviewers, and make sure that recommendations made are considered when the manuscript is revised by its author(s) (the role of the academic coordinator is further discussed below). Thus, although authors/editors may be involved in suggesting the academic coordinator, it is the coordinator who identifies suitable, independent reviewers, and the review process is either single or double blind. While the ambition in most cases is that authors and reviewers should be unknown to each other (double blind) it remains difficult to uphold the anonymity of authors — and possibly also of reviewers — as the social sciences and humanities community in Sweden is rather small. Still, academic coordinators and reviewers do not need to be — and often are not — Swedish, but obviously it is required that they are literate enough in the language of the manuscript to be able to assess it. Hence, international reviewers are often used for English language manuscripts and for contributions in Swedish it is not uncommon with coordinators or reviewers from Denmark or Norway. After one or two rounds of review, the coordinator will write a recommendation to the board of Kriterium, who then makes the final decision on acceptance. This process and the actors involved are schematically illustrated below 5.Generally, but not always, the review reports are structured according to the instructions given to reviewers. The interface for submitting the reports reflects the criteria used for assessing the quality of the manuscript, consisting of the following headings: “Knowledge claims and themes,” “Structural and linguistic clarity,” “Method-theory-empirical data/sources,” “Scientific context,” “Recommendation and summary of proposed changes,” and “Detailed comments and suggestions.” In addition to these headings, reviewers are given the opportunity to write comments directly to the authors and the publishers, and they are also asked to anonymize their report and disclose any “competing interests.” Despite this structuring of the review process, there is great variance both in terms of length and focus in the reports. The same applies to the style; a few reports show similarities to a published book review, whereas other reports follow the structure envisaged above more closely. These observations correspond well with earlier findings which suggest that instructions for reviewers have little influence on how the actual reports are written .An overarching question that emerged from our reading of the material was how reviewers and academic coordinators relate to the function of peer review in terms of gatekeeping and/or feedback and with attention to various audiences. In the following, we will focus on three themes that shed light on this broader question through the localized example of Kriterium. The first theme concerns the intended audience and how reviewers navigate possible tensions between intra-academic norms and accessibility for a general audience. The second theme addresses how peer reviewers approach the task of assessing edited volumes; here we find the navigation between the parts (chapters) and the whole volume particularly interesting. The third theme reflects on the role of academic coordinators and reviewers and highlights different interpretations of what these roles entail.The first theme identified in the analysis of reviewer reports concerns questions regarding the intended audience of the scholarly monograph. Of particular importance in this discussion is how to find a balance between a more popular style of text for a general audience and a more scientific, potentially inaccessible style. One reviewer writes: “It is possible that this way of writing scientifically works in a PhD dissertation, but it does not work as well in a book directed to a general public interested in the topic” (Mon D: Rev). In a similar vein, reviewers often comment that authors “give in to the temptation” of offering too much “historical detail” (Mon C: Rev), and it is suggested that meticulous and lengthy accounts may limit the audience to a “local  readership.” [Mon A: Rev ] Evidently, the potential audience is an important parameter in the assessment of monographs. However, targeting an academic audience as well as the general public could possibly come into conflict with other, more intra-disciplinary, ambitions. These tensions are expressed by one reviewer when writing: “The book appears as a work of scholarship where strengthening the argumentation is perhaps more important than to communicate with the general public.” (Mon D: Rev).Against this background, it comes as no surprise that comments pertaining to style and rigor are prevalent in the reviews. This criterion even has its own heading, “structural and linguistic clarity,” in Kriterium’s instructions for reviewers. Sometimes the text is criticized for being too “heavy” and not very appealing for a general audience (Mon D: Rev), yet there are also instances where the presentation is praised for being “accessible and confident” (Mon K: Rev). At the same time, reviewers put a lot of emphasis on scholarly rigor, careful handling of sources and appropriate referencing. This is of course highly important for a general public because they may be in a less informed position to identify flaws in the text, but an extensive use of references generally characterizes the academic work rather than the popular work of non-fiction. In fact, one of the manuscripts received substantial criticism with regards to referencing, as the reviewers suggest that it was difficult to discern the author’s own contribution in relation to previous literature. Critique was also raised against using translated sources, rather than the original, which was described as “deviating from common scholarly practice” (Mon E: Rev). This suggests that the perspective of the reviews is often primarily a scholarly one, but with an eye towards a broader market.Generally, in the books studied, reviewers have been recruited from the same field as the author(s), or a closely related one, but several books are of a multidisciplinary nature and there are instances involving reviewers with different disciplinary background. In one case, the reviewer comments on this at the beginning of the report, noting that the comments will be from the reviewer’s disciplinary perspective rather than that of the author (Mon F: Rev). In negotiating disciplinary differences, both before reviewers accept the task and when authors respond to the comments, the role of the academic coordinator is central. An important responsibility of the academic coordinator, much like that of journal or book editors, is thus to be a link between reviewers and authors. This is especially important when handling processes involving several disciplines or schools of thought. Such a task might be challenging with regards to monographs with interdisciplinary content, and even more so for edited volumes that may include chapters from a range of fields, where it is important that each chapter is assessed on its own disciplinary principles. This is expressed by a coordinator when writing their recommendation: “A premise in academic assessment is that each chapter is reviewed within its own paradigm. In the assessment of (author’s) chapter, I believe that the reviewer and the author are part of different paradigms.” (Edv D: AC) When addressing shortcomings in peer review, especially of edited volumes, the responsibility of an editorial function is thus visible in the material.Consequently, the multiple roles of the scholarly book as simultaneously an intra-disciplinary contribution and disciplinary achievement (merit), a work expected to be read by scholars from other fields, and a means to communicate broadly both within academia and outside it has consequences for how peer review is performed. A balance between scholarly rigor, methodological and theoretical refinement, and an accessible presentation appears to be a key quality sought after in the assessment. Another balance, between parts and whole, is an important theme in the next section which looks at the reviewing of edited volumes.The edited volume consisting of chapters written by different authors on a specific theme or topic is a popular genre, especially in the humanities. In the social sciences and humanities as a whole, book chapters are the second most common publication type , only preceded by journal articles . The ediThe Kriterium review reports of edited volumes, as in the case of monographs, vary in length, focus and adherence to the review topics. As can be expected, a large part of most review reports concerns such aspects as interpretations, previous research, methodology, and theory. However, there are also recurring comments on the format of the edited volume itself that reveal some ambiguities towards quality control and improvement. Often commented on in the reviews is the heterogenic nature of the edited volume. Suggestions for how the manuscripts can better come together as a whole can be found in a majority of the reviews. This lack of coherency has been dubbed the “most common weakness of the edited volume” by Edwards . The basic tension between the whole and its parts is summed up in the following comment in relation to a reviewed manuscript:Detailed comments below, with, in some cases, suggestions of improvement for the benefit of the anthology as a whole (Edv L: Rev).Since this is an anthology with more than twenty different contributions, there are variations in the way the treated subjects are approached, from more personal reflexions (sic) to solid and original research. Therefore each article is commented (on) individually in the section The quote illustrates some of the challenges of cohesion that reviewers are faced with, even though some reviewers are careful to point out that they believe the plurality of methods and/or perspectives to be beneficial to the manuscript as a whole . Nevertheless, despite the fact that diversity is arguably built into the genre, many reviewers clearly expect an edited volume to be focused on one topic or theme. One reviewer openly discusses the “challenge” of putting together an edited volume in that individual case studies need to be brought together in a way that makes the volume appear coherent (Edv D: Rev). Interestingly, one reviewer begins with a personal and general observation, claiming that the genre is becoming less prominent with international publishing houses within certain subjects of the humanities and social sciences see also . AccordiOther aspects of this theme deal with differences in quality or scientific rigor. One reviewer even suggests that some of the contributors to a volume under review might be looking for a “shortcut” to publishing even though the quality is not up to scientific standards (Edv A: Rev) and adds:This book aims at collecting the authors under one common aim and that must, in my opinion, be made clearer in each contribution. As it stands, it is transparent that some of the authors of the individual chapters have a different purpose with their contributions than the one for which the editors have brought them together.How, then, can lack of cohesion be remedied and coherence be achieved? A common way of correcting the sense of incoherence is to suggest revisions of the introductory chapter, described by one of the academic coordinators as “a key chapter to legitimize the volume as a unified, coherent publication” (Edv D: AC. Illustrative examples include a remark that the introduction “needs to be rewritten so that it lays out the point of the book more clearly” (Edv I: Rev) or that “the theme of the book needs to be reconsidered” so that the “common aim of the chapters” becomes clear (Edv A: Rev). One reviewer elaborates on the theme:This is my greatest reservation about the manuscript. It it (sic) possible that the material might be more suited to two theme issues of a journal. While impressive efforts have been made to frame the material as cohesive in the title and in the Introduction (Ch. 1), Parts I and II do quite different things (Edv G: Rev).Other suggestions include practical changes to the overall structure, including moving chapters around or changing titles . Individually, authors are at times also encouraged to reference the common topic or theme more clearly or, as in one case, to “align the methodology of this chapter with the stated aim of the book as a whole” (Edv G: Rev). More drastically, one reviewer suggest that three chapters get merged into one in order to improve “structural clarity” (Edv L: Rev). Many reviewers suggest formalizing language, concepts, and tone as well as cross-referencing as ways of achieving a more unified product.A more radical approach to rectifying lack of cohesion is to suggest the exclusion of certain chapters, due either to lack of quality or thematic connection. One might argue that rejection is indeed a fundamental part of journal peer review but can of course present problems for the editors of an edited volume that is the result of a conference or an initiative among local colleagues. This is put in no uncertain terms by one reviewer via the maxim that a chain is never stronger than its weakest link (Edv M: Rev). In a majority of the fourteen volumes discussed here, at least one of the two reviewers suggests that an individual chapter does not fit the volume’s scope, with several reviews signaling that chapters should be dropped prior to publication. In one particularly critical review, the three initial chapters are said not to “meet the standards” (Edv K: Rev). A couple of reviewers are even so adamant that they cannot recommend publishing unless the chapter is dropped . Others recommend that the author in question should consider alternate publishing outlets . Remarkably, in one review process, however, the two anonymous reviewers suggest different chapters to be excluded (Edv G: Rev). Still, in the cases analyzed here, no volume seems to have actually eliminated a controversial chapter, instead opting for major revisions to the chapter in questionAs shown, the reports reveal points of ambivalence in how to accurately review and revise edited volumes. In some instances, these tensions are discussed more directly in the reviews. One very positive reviewer still claims to “sense some ambivalence in how the project has been put together”  and another proposes that the geographical and institutional background of the contributors needs to be described and motivated “in the introduction in order to avoid the suspicion that the volume adds nothing more than being a  an existing theoretical framework” (Edv J: Rev). A third example is one reviewer who maintains that although not all of the information provided by the book is “totally new”, this could “indeed hardly be the case in this kind of publication” (Edv L: Rev). Interestingly, one reviewer wants the edited volume to “dare move away from the traditional article” in order to become more appealing for a general audience (Edv A: Rev). Clearly the edited volume poses a challenge for reviewers. These difficulties may be partly attributed to reviewers addressing their comments both to individual authors of chapters and to the editor(s) of the volume. The editor(s) often acts as a link that mediates reviewer comments to the authors, thus having considerable influence on how feedback is received and communicated. In the review process developed by Kriterium we also find another intermediary who oversees the process and the communication between authors, editors, and reviewers. The role of the academic coordinator is the main theme in the next section.The academic coordinators have a somewhat unique role at Kriterium. Although they can be said to perform an editorial role, they generally only do so once and in relation to a manuscript within their field of expertise. They are usually suggested by the publisher (presumably in consultation with the author or volume editors) at the time of submitting a proposal to Kriterium. The Kriterium editorial board approves the choice after ensuring that there is no tight personal or professional link between the actors involved. An exception may occur when the publisher is a university publication series, in which case the series editor, who may be a colleague of the author’s, often takes on the role of academic coordinatorThe work of the academic coordinator is largely similar to that of a journal editor or publisher editor, in that they secure reviewers for the manuscript and ensure that the reviews are received. Moreover, the academic coordinator provides an independent assessment of the reviews based on their own reading of the work in order to advice the authors/volume editors on how to address any necessary revisions of the manuscript. There are several examples of academic coordinators expressing that they take this advisory role seriously, for instance:The reviewers state fairly different opinions on the manuscript. Both reviewers recommend that it be published, but with minor revisions. In this review, I go through the two review reports and provide a recommendation on revisions the editors should or can consider. Thus, I leave a door open if the editors have very diverging opinions (Edv M: AC).In the context of academic book publishing, a shift has been identified from the role of the editor as gatekeeper to a more proactive role as commissioning editor, as the editor is also tasked with analyzing business opportunities and recruiting authors to write books that address those opportunities . ContrarThe academic coordinators display varying understandings of their role and these interpretations in turn affect how they interact with the authors or volume editors. The relationship with authors and editors is formally similar in that the academic coordinator generally does not (and should not) know the book contributors . The interaction is generally initiated after the reviewer comments have been received and the academic coordinator communicates the reviews to the authors or volume editors, along with their own interpretation. A text on the Kriterium website describes the relationship between academic coordinator and author as: “When reviews are completed, the author implements changes to the manuscript in consultation with the academic coordinator.”the collaborator, illustrated by an academic coordinator’s report which describes how “contacts have been frequent between the editors and the academic coordinator” (Edv C: AC) to formulate instructions to the chapter authors and ways of communicating revisions. In some cases, the academic coordinator even proposes solutions to the problems identified by the reviewers:The reports to the Kriterium editorial board indicate that some academic coordinators take quite an active role in the work that follows, not least in the case of edited volumes. This is an interpretation of the role we term This can also (perhaps in combination) be achieved by more work on the introduction and a division of the various chapters — perhaps with short intros to each part. I couldn’t help but test it, and think about how one could do it, I don’t know if I go too far in my role as academic coordinator but I choose in this letter to make a suggestion (mostly as an example of how one could do — not an example of what one should do) (Edv J: AC).In another report, the academic coordinator indicates having discussions and sharing concerns with the volume editors with regards to some of the book chapters: “Both I and the editors thought that there were problems with the citation technique” (Edv L: AC). There are occasionally similar descriptions of ongoing contact and discussion between the academic coordinator and the author of a monograph.project manager than as a constructive partnership. This is the case, for example, when the academic coordinator asks the editors to propose a “strategy and time plan for the revision of the manuscript” . The line between the collaborator and project manager roles is sometimes blurred, as seen in the following quote where the process of “re-writing” the book manuscript is described: “We decided to have a Zoom conference … to discuss the introductory chapter. And as academic coordinator I must applaud the substantial re-writing ...” (Edv J: AC). Notably, the coordinator in this case see themselves as part of a “we” discussing the revisions, yet at the same time the role is also that of someone making an assessment of the process (applauding).In other cases, the academic coordinator seems to take on the role more in terms of a an examiner, they are more inclined to provide direct requirements for what they expect to be revised in order for the work to achieve sufficient quality for Kriterium. For instance, one academic coordinator report supplements the reviews with very clear instructions: “This added chapter is a MUST on my part, no matter how well-written R2 considers your current final remarks to be.” (Edv K: AC).When the academic coordinators interpret their role as The academic coordinators taking on the examiner role often provide constructive suggestions to the authors or book editors of how the work may be improved, but do so as a requirement rather than by inviting dialogue. A more extensive example is offered below:Make sure the authors limit their references to all kinds of likely and unlikely idols but please encourage them to thoroughly read the theoretical sections (chapters) in (xx’s) dissertation. Ideally, these would impact the analytical approaches in separate chapters as well as in your discussions (Edv K: AC).Such requirements as exemplified above occur both at the stage when the academic coordinator’s comments are sent to the authors or volume editors and when academic coordinators report to the Kriterium editorial board on their evaluation of the revisions made. In the latter case, however, the required changes are generally described to be “things that can quickly be remedied” (Mon G: AC). Even so, there are also examples where a “re-write and re-submit” procedure is recommended based on the reviews.It is reasonable to assume that for some of the reviewers and academic coordinators, the role as reviewer of a book pre-publication is a new experience. Unlike the constructively critical role of a scholarly seminar participant or dissertation supervisor, the reviewer and coordinator partly take on the role of gatekeeper, in that they advise the Kriterium board on whether or not to accept the manuscript. Similarly, there are several examples of reviewers who with little hesitation suggest that a manuscript must be revised (Mon J: Rev) or even rejected (Mon E: Rev), and an important part of the reviewer task is to conclude if a manuscript is acceptable, in need of required revisions, or if it should be rejected. Yet, another role, that of a discussant, or critical friend, also comes across in the reports. In a couple of reports, the reviewers actually discuss their role and the function of peer review in relation to how their recommendations should be viewed:My intention is that the author should be provided an opportunity to reflect on these suggestions in order to facilitate certain changes. However, without in any way advising against publication or require a new review round. The discussion, even based on the current manuscript, may of course also contribute to the process of increased knowledge and understanding of the questions raised by the study (Mon E: Rev).Similar views are expressed by an academic coordinator for another monograph:At the same time the author cannot be criticized for making certain choices; it would be unreasonable to ask for a different book . The peer review should be read as suggestions for revision, and not as a rejection of the project (Mon D: AC).In our view, these statements can be related to the integrity held by the “author” in parts of the social sciences and humanities. Indeed, it could be argued that the scholarly author — especially in fields were monographs play a central role — is seen as an individual and unique subject (much like a fiction author). Hence, suggestions for major revisions may threaten the autonomy of the researcher. Reviewers as well as academic coordinators may therefore be cautious when formulating their recommendations.enhancing and certifying quality — is a recurrent theme throughout the material.As the descriptions above have shown, although the primary function of the review process is that of gatekeeping, of ensuring high academic quality, it is also in many cases viewed as a quality-enhancing and supporting process. Several of the academic coordinators express great appreciation at how seriously the authors and book editors have taken the task of revising and how thoroughly they have worked with their texts based on the reviews see also . In the The focus of this paper has been on the review process of Kriterium and how it serves to achieve the twin goals of quality enhancement and quality certification of scholarly works. With an eye towards the contested role of the scholarly book that prompted the development of Kriterium in the first place, we have looked for instances in the review process where inherent tensions existing in publishing social sciences and humanities research might become visible, thus contributing to the knowledge about peer review in these disciplines. The first tension arises in relation to the intended audience of especially monographs in the humanities, which are expected to address both a scholarly and a general audience. Hence, in adhering to these expectations a work should both have scholarly rigor (to be given the criteria of “scholarly”) and, at the same time, be of interest to a broader audience. Many of the comments made by reviewers and coordinators, especially on Swedish-language monographs, reflect on this tension, where finding the balance between scholarly achievements in terms of theoretical and methodological proficiencies and an attractive and accessible presentation is of great importance. While we find that scholarliness (including disciplinary jargon) and accessibility may be positioned in opposition to each other, it is also evident that many reviewers and academic coordinators do not see these as mutually exclusive. Indeed, a high-quality monograph in many humanities fields displays high scholarly quality in a style that can also attract non-expert readers.Contrary to monographs, edited volumes primarily target a scholarly audience and they are more often written for an international audience. In our analysis we find that a particular concern raised by reviewers is how the parts (chapters) relate to the volume as a whole. Typically, discussions involve matters such as how the volume holds together , and whether or not all chapters have the quality required to be included in the volume and eventually marked with the Kriterium quality stamp. Interestingly, reviewers are concerned both with individual chapters and the volume as a whole. This marks a difference from many international publishers, where chapters often are reviewed in isolation. By using the same reviewers for the whole volume, Kriterium puts emphasis on the coherence of the book. While such an approach could possibly result in more integrated and coherent edited volumes, it may also give rise to specific tensions in the review process. For example, there are several examples of reviewers suggesting that specific chapters are thoroughly revised, or even dropped from the edited volume completely. However, from what we have observed, chapters are rarely omitted, although they may be revised. This could be viewed as a confirmation of the notion that edited volumes allow for contributions of varying quality and relevance. Such an interpretation would be in line with collaborator, who will work actively to provide suggestions for improvements, thus contributing to enhancing the quality, as a project manager, who tries to facilitate the process of finishing the work, and as that of an examiner, who takes on the role of quality gatekeeper. These are all roles that could characterize the book editor, but with the important difference that the academic coordinator performs the task once, and with a focus on quality assurance only, without considerations for the other aspects of book publishing. Notable among both reviewers and academic coordinators is the respect shown for the choices made by authors, which we suggest may be a consequence of the relative status, autonomy, and independence of the author in many humanities and social sciences fields.The academic coordinators at Kriterium play an important role in mediating between authors/volume editors, reviewers, and the Kriterium editorial board. These coordinators perform functions that are similar to an editor of a journal in that they select reviewers and facilitate the review process. Yet, the analysis makes evident that the coordinators sometimes interpret their role in partly different ways from what is generally expected from a journal editor, or even book editor. The coordinator reports illustrate that the academic coordinators interpret their task as being that of a In a broader context, Kriterium projects questions about academic publishing patterns. Whereas the social sciences in Sweden have more wholeheartedly embraced the international tradition of peer-reviewed journals, impact factors, and “publication points”, the humanities in Sweden, as in many other countries, are still characterized by more pluralistic patterns of publishing . To be sThe peer review of scholarly books, both monographs and edited volumes, is still a novel practice within Swedish social sciences and humanities. In light of its origins, Kriterium, interpreted as a response to shifts in academic publishing and their perceived consequences, in many ways represents anxieties present in the humanities today about the future of publishing. The peer review of scholarly books that is at the core of Kriterium, can be seen as a means to an end — to validate and protect the status of scholarly monographs (and edited volumes) — rather than an objective in its own right, even if the value of quality control certainly was part of early negotiations. In retrospective interviews, many participants in the pilot phase voiced their conviction that “the humanistic tradition to write monographs needs to be valued higher” , 8. Inte


This represents a big leap ahead (+ 52%) compared to the IF of 2019: 4.797, and it is the highest IF ever recorded for a headache journal [The Journal of Headache and Pain as the first headache journal, in the first quarter (Q1) for the category of Neurosciences—where the journal now ranks 36th out of 273 journals—and in the Q1 for the category Clinical Neurology .On the 30th of June 2021, Clarivate Analytics announced the release of 2020 Journal Citation Reports (JCR), announcing the 2020 Impact Factor (IF) of We are aware that this year Clarivate changed how it calculates IFs, and this will have had some positive effect. This year’s IF traditionally would have considered citations in 2020 for content that was published in issues in 2018 and 2019. This year Clarivate adjusted the ‘cut-off’ point for inclusion of content, from the issue date to the date of online publication for content published from 2020, thereby increasing the number of considered citations in the new IF calculation  . While tAll credit goes to the Co-Editors, Associate Editors, Advisory Board, and Junior Editorial Board and also to our Reviewers and Authors for the choice of content that has been published in our journal and for their contribution to the journal growth since its foundation . Thank yWe are deeply grateful for the help and support of the whole team, and we are proud of this important and “unbelievable” achievement that we owe to them. However, although many have used the term “unbelievable,” I prefer to say “ensuing” global success. This achievement is a result of a tight and continuous team working together, sometimes sailing upwind, close-hauled.Of course, citation is not the only measure for high impact; thus, I am more than happy to acknowledge that the journal has been growing over the years in quality, volume, and reach as well. Submissions increased by 28% in 2020 vs 2019 (426 submissions in 2020) and are peer-reviewed with excellent turnaround times: on average, it takes 17 days from submission to first decision and 68 days from submission to acceptance. Publications have increased over the years, and in 2020, 136 articles were published in the journal (+ 20% vs 2019). Full-text downloads increased astoundingly in 2020  along with the number of shares on social media.Much is also due to all those who said many years ago that a new journal in the field of headaches “could not be done” and that “it should not have been done,” and we also owe a lot to those who said that “they would have never applied for an article processing charge”—enough to understand that we had to continue developing the project.A multivoiced scientific community is more authoritative, and it tends to compete fairly and improve itself.It is also interesting to note how once again all the headache journals have increased their IFs, witnessing the moment of good health that our scientific community is experiencing. We are no longer considered as the “Cinderella of neuroscience.” Of course, a significant boost to this cultural growth can be attributed to the new classes of dedicated drugs that are still struggling but progressively permeating migraine therapy. Certainly, another impetus is given by the enormous amount of educational activity carried out in recent years by scientific societies in the training of new recruits, who are already showing their scientific skills as promising rookies.We know that there are no immutable situations, even if guaranteed by mandatory citability privileges, and we will work to consolidate this result, opening ourselves to new ideas and connecting with advanced areas of biomedical research in headaches with our open access contents, without leaving out emerging areas. Our perspective will continue to be multidisciplinary, given that headaches are a topic of study from different perspectives and specialties; scientifically, they are not a privilege of the few. Young people will continue to receive easy access through successful initiatives such as the EHF School of Advanced Studies Reviews.However, we continue to believe that headaches must also be clinically tackled on the vast platform of clinical medicine, and for this reason we tend to operate on multiple fronts.We are fortunate to have on our side a publisher who is rigorous on quality, prone to new ideas, striving toward innovations, anchored to best practices in publishing, and in any case always respectful of the scientific choices made in the selection of the works to be promoted.The projects in the pipeline are many. Several thematic series are still open for submissions, such as the one on Translational Research in Headache and Basic Science in Headache. A thematic series has just been opened to collect all papers that over time won the Enrico Greppi Award, traditionally hosted here. New exciting Thematic Series will be launched soon.A refreshment of the Editorial Board in the forthcoming months will allow us to welcome those who volunteered to join our team to help us build a competitive, fast, independent “headache-research world” that is constantly renewed in quality and dedicated to what we have chosen over time, the care of headache sufferers.


Opportunities for early career researchers (ECRs) to engage with the peer review and publication process can be few and far between. Last year, we launched a pilot support programme to introduce ECRs to peer review. PRE-review Open Reviewers pilot programme aimed to address this gap by providing formal training material to ECR mentees and journal editor mentors who then engaged in reviewing preprints in pairs. But usually, if they are lucky, ECRs can learn on the job by performing a review under the supervision of more experienced colleagues.Early career researchers (ECRs)—PhD students, post-docs as well as recently-appointed principal investigators—constitute a large fraction of the scientific community, however they are often excluded from direct interaction with the publishing process, particularly with regards to reviewing the work of others. Frequently, there is no formal training available on how to become a fair and constructive peer reviewer. The “82% would recommend the programme to another researcher”Nature Reviews journals initiated a programme to encourage reviewers to enroll and mentor an ECR from their group through the peer review of an article, while providing at the same time a framework to formally recognise the work of the mentee.Even in that case, it is not unusual for the ECR to not receive credit for the work performed as a reviewer. To address this issue, in 2020, the However, for the most part, the situation is frustrating for the ECRs, who feel left out of the publishing system. This cohort of investigators will lead the future of research but currently the peer review process is not benefiting from their diverse contributions as much as it could to shape advances within their fields.Sense about Science, a UK-based charity that “champions the public interest in sound science and ensures evidence is recognised in public life and policy making”. The programme started with two webinars; the first one delved into the need for peer review, including its importance for research reporting and scientific progress, while the second discussed the specifics of editorial processes and gave practical guidance on how to be a fair reviewer and provide constructive suggestions aimed at improving the technical soundness and rigour of the conclusions presented.In an effort to make peer review more inclusive for those investigators, in 2020 we piloted a support programme for ECRs in collaboration with Voice of Young Science network. About 30 of those investigators were selected to participate in a hands-on phase in which they received the direct support of our editors. Each participant was paired with one of the editors at Nature Communications, based on their field of expertise. The editor then invited the ECR to peer review a relevant article submitted to the journal, in addition to the usual set of reviewers sought for each paper. This was an opportunity for the participants to put into practice the advice received during the webinars. Upon return of the reviewer report, the ECR received feedback from the editor on the quality and usefulness of their comments, and further suggestions on how to write a high-quality reviewer report.The webinars were attended by about 200 ECRs from the A survey of the participants showed very positive results: 73% said that the feedback received from the editor had been helpful or very helpful, 91% agreed that the programme gave them more confidence to review papers, and 82% would recommend the programme to another researcher. Many commented that the programme not only gave them the toolkit for peer reviewing, but also more generally supported their confidence and professional development.The ECRs particularly valued the opportunity to review the work of others in a safe space, and would have been keen to review more papers. We identified areas of improvement in the programme that will ensure a better expertise match of the participants to relevant papers.The small-scale pilot has been useful to test the logistics of the programme, and this year we want to offer support to as many ECRs as we can, focusing in particular on those who otherwise have more limited access to journals and editors—for example, if their institute is not located where in-person conferences usually take place.apply!Having built relationships with this cohort of new reviewers, we look forward to meeting and engaging with many more future and diverse research leaders this year. We are starting enrollment now and the programme will start in June—if interested, please


Dear Editor:International Journal of Colorectal Disease by Schietroma and co-workers [With great interest I read the article “Mid-and low-rectal cancer: laparoscopic versus open treatment – short and long-term results. Meta-analysis of randomized controlled trials” recently published in the November issue of the -workers . I apprep = 0.114), but it implies again a benefit of the laparoscopic approach, as impressively demonstrated in the Kaplan-Meyer survival curves presented in the original article published by Ng and colleagues [This latter statement is presented as a result of a meta-analysis of four trials that have investigated long-term DFS, as presented in table 11 of the article –5. LookiThus, not even one of the four trials processed in the meta-analysis indicates a trend towards inferiority of the laparoscopic approach but all rather indicate equivalence if not superiority. Accordingly, the pooled data of the studies as presented in table 11 of the authors’ manuscript demonstrate 79% (409/518) long-term DFS for laparoscopy and only 74% (410/553) for open surgery. It is hard to understand how the authors could come to the result that “4 and 5 years disease-free survival were statistically higher after open surgery.”It is a pity that this wrong statement is presented as a key message of a peer-reviewed article in a well-known international journal. The statement is likely to be cited in the future frequently as evidence for oncologic inferiority of the laparoscopic TME, although it obviously results from an error in the authors’ analysis.


The qualification and order of authorship in scientific manuscripts are the main disputes in collaborative research work.The aim of this project was to develop an open-access web-based platform for objective decision-making of authorship qualification and order in medical and science journals.The design science process methodology was used to develop suitable software for authorship qualification and order. The first part of the software was designed to differentiate between qualification for authorship versus acknowledgment, using items of the recommendations of the International Committee of Medical Journal Editors. The second part addressed the order of authorship, using the analytical hierarchy process for objective multiple criteria decision-making and ranking. The platform was evaluated qualitatively (n=30) and quantitatively (n=18) using a dedicated questionnaire, by an international panel of medical and biomedical professionals and research collaborators worldwide.Authorships.org represents an open-access software compatible with all major platforms and web browsers. Software usability and output were evaluated and presented for 3 existing clinical and biomedical research studies. All 18 international evaluators felt that the Authorships.org platform was easy to use or remained neutral. Moreover, 59% (n=10) were satisfied with the software output results while the rest were unsure, 59% (n=10) would definitely use it for future projects while 41% (n=7) would consider it, 94% (n=16) felt it may prove useful to eliminate disputes regarding authorship, 82% (n=14) felt that it should become mandatory for manuscript submission to journals, and 53% (n=9) raised concerns regarding the potential unethical use of the software as a tool.Authorships.org allows transparent evaluation of authorship qualification and order in academic medical and science journals. Objectified proof of authorship contributions may become mandatory during manuscript submission in high-quality academic journals. Next to the presentation of scientific results, authorship in academic journal articles is a means for scholars to communicate the intellectual contributions of their work, take public responsibility for it, build reputation among peers, and convey their professional benefits . HoweverTo address these issues, the International Committee of Medical Journal Editors (ICMJE)  establisWith the interest of promoting the highest ethics in medical and science publications, the aim of this project was to develop a user-friendly, open access, web-based software platform for the objective assessment of authorship qualification and order, in an attempt to reduce and hopefully eliminate authorship disputes.The design science research methodology was used to develop a suitable software for achieving our objective, as previously described . BrieflyThe software was designed by DAR located in Switzerland and developed together with AR located in Athens, Greece, starting on May 7, 2016. A group of collaborators worldwide contributed in the design, development, and evaluation of the software, and were selected based on personal contacts and through personal interest, with a full list of names available in the Acknowledgments and on Authorships.org. A subset of 18 of these collaborators were selected for formal software evaluation as described below, with eligibility for selection being a scientific background in medicine or other biomedical areas, regardless of the grade. The characteristics of the collaborators who formally evaluated the software can be found in The free web-based software is built on Drupal version 7 , an openThe first part of the software design focused on identifying and differentiating individuals whose work qualifies them as authors as opposed to contributors for acknowledgment. Items of the International Committee of “Medical Journal Editors Uniform Requirements for Manuscripts Submitted to Biomedical Journals,” including conception, execution, and documentation of research, were used to design the first part of the software  4]..4].The second part was designed to address the order of the qualifying authors using the analytical hierarchy process (AHP) method for objective multiple criteria decision-making and ranking . BrieflyThe software was evaluated and re-evaluated during the design phase as described above. At the end of software development, 18 collaborators  assessedThe developed software, named Authorships.org , is freeFirst, the user is requested to add the number and names of the individuals who participated in a particular research project. Next, their contribution to the work is assessed based on the generally agreed upon ICMJE criteria, such as conception, execution, documentation, and final approval for publication  4]. The. The4]. Based on the existing ICMJE guidelines, the software then indicates which individuals qualify for formal authorship versus for acknowledgment as a contributor. The user is subsequently asked to indicate a senior/last author of the manuscript, if required .In the second part, the user assesses the extent of the individual contributions of all qualifying authors, except the one defined as senior/last author. A ranking of each author’s extent of contribution to each ICMJE criterion is performed by using the AHP multiple comparisons method. Thereby, each author is compared to each other author in pairwise comparisons, using a decision matrix for ranking of the extent of contribution .After all pairwise comparisons are made, a consistency ratio is provided to the user to ensure the input data are consistent and logical . This raBased on these quantitative and objective evaluations, the software defines the appropriate order of author appearance in a publication at a medical or science journal .Eighteen international collaborators evaluated the software using the specially designed questionnaire. Of these, 88% (n=15) felt that the software was easy to use, while 12% (n=3) remained neutral. Additionally, 59% (n=10) were satisfied with the software output results, feeling that it objectively reflects reality, while 35% (n=6) were unsure and 6% (n=1) were dissatisfied. The reason stated for dissatisfaction was that the collaborator felt “one very important contribution  may be sufficient to qualify for authorship;” however, authors with such contributions were disqualified when using Authorships.org. Fifty-nine percent (n=10) stated they would definitively use the software in future projects, while 41% (n=7) would potentially consider it. Moreover, 94% (n=16) felt that it may prove useful to eliminate disputes regarding authorship, and 82% (n=14) felt that it should become mandatory for manuscript submission to journals. Fifty-three percent (n=9) raised concerns regarding potential unethical use of the software as a tool for authorship evaluation. Reasons given were concerns that Authorships.org may represent one more tool to “justify” unethical authorship behavior. The quantitative results are summarized in In an attempt to objectify authorship disputes, 6 collaborators contributing to a clinical study were asked to provide the authors’ order and reasoning behind it for a manuscript not yet submitted. The results of this survey were analyzed in a particular form developed at Authorships.org. Although they all agreed 100% regarding the authorship qualification of all 6 collaborators, a 0% total agreement on the order of author appearance was noted . In retrospect, the authors felt that the main reasons for this discordance were misconceptions on the amount of work conducted by their colleagues, lack of adherence to the “International Committee of Medical Journal Editors” recommendations, and some form of bias. Authorships.org was then applied to review the authorship order, yielding complete agreement of the collaborators. The collaborators felt that the platform was a transparent and objective tool for assigning authorship order, and justified the use of Authorships.org for their work.Seven authors of a biomedical engineering manuscript based on clinical and medical imaging data were asked to provide the order of the authors for the paper they all contributed. All authors agreed on their own qualification for authorship, and 2 (28%) authors agreed about the collaborator order for the manuscript. Authorships.org was used to assess author qualification and the order of authorship. Thereafter, 5 out of the 7 authors justified the use of Authorships.org and thought that it is a robust and objective tool to assign the order of authors. The reason for the disagreement of the remaining 2 authors was a feeling that Authorships.org did not sufficiently acknowledge the seniority level of individual authors. They further disagreed to the design that authors should fulfil at least one green item, one red item, and the blue item to qualify for authorship in Autorship.org (based on the ICMJE recommendations) .Eight members of an international research group conducted a systematic review and meta-analysis regarding the use of comprehensive enhanced recovery protocols in the setting of liver transplantation. Individual tasks included study design and methodology, screening of titles and abstracts of studies identified in the literature search, data extraction and analysis of selected works, manuscript writing, and critical review. Authorships.org was applied after drafting the manuscript and before submission to a peer-reviewed scientific journal. The objective author qualification and order were applied directly in the manuscript as suggested by the software output and accepted by all authors.This study reports on the development and evaluation of a novel software tool for the objective assessment of authorship qualification and order, in an attempt to reduce authorship disputes. To the best of our knowledge, Authorships.org represents the first open-access web-based software to quantitatively and thus objectively indicate the qualification and order of academic authorship in medical and science journals.The majority of collaborators assessing the usability of Authorships.org found the software helpful and easy to use, and felt that the software could have a high impact in the scientific community in a short period of time. The concerns of the remaining collaborators regarding software output results focused on the criteria required for authorship qualification and therefore on the internationally accepted ICMJE criteria applied in the proposed software algorithm. While the qualification for authorship has been standardized in the ICMJE criteria , varyingOne main hypothetical advantage obtained by the implementation of an objective instrument in academic authorship, as presented herein, is a reduction in the number of disputes. Academic authorship of papers arising from complex research projects, involving multiple centers and with potential valuable impact on the scientific community, can nowadays be considered as a currency for career development. These scenarios represent feeding terrains for conflicts between authors, which can sometimes even result in the retraction of a manuscript from publication or, if remaining unresolved due to failed mediation among authors, in rejection by the journal. Another known issue is the abuse of power exerted by certain authors inflicting the order of authors in their own interest, which can be avoided by using an objectifiable tool such as Authorships.org. We believe that the most reasonable attempt to resolve such disputes is undertaken primarily among the authors themselves, and in case of failure to compromise, the disputes are taken up by the authors’ affiliating institutions. Moreover, Authorships.org allows authors to evaluate the extent of contribution provided by each author to individual phases of manuscript preparation, representing the main advantage of using the AHP over other ranking models. Another main advantage of the proposed algorithm is the possibility to “customize” the relevance of each area of performance according to the contributors’ previous agreement or the institution’s internal guidelines. The use of the software could prompt a prearranged settlement and, consequently, help the team working on the project on the basis of mutually approved rules . This further allows maintaining the “motivating” aspect of a researcher to perform at her/his highest level when aiming to keep a certain position within the authorship list. Most of the collaborators evaluating the software further felt that the added consistency ratio supported them in reflecting their own contributions to a project in a quantitative way. A progressive improvement in consistency ratio values with increasing use of the software is further expected, as a result of ongoing education on the use of a mathematical way to evaluate each participant’s contribution.prior to the start of a research project, Authorships.org was deliberately designed to be applied after the execution and writing of a manuscript but before submission to a scientific journal, preferably in a discussion by all contributing members of the research team [An important aspect of the Authorships.org design is the time point of the assessment of the qualification and order of authorship. While in many research groups worldwide, author roles are defined rch team . Hence, In conclusion, Authorships.org represents a novel approach to quantify and objectify the qualification and order of authorship in academic literature. It may become a mandatory tool for objectified proof of author contributions in scientific publications. Further randomized studies are needed to validate the potential of using such a tool for reducing or eliminating authorship disputes.


We identified 651 research outputs on the topic of COVID‐19 in the form of preprint, report, journal article, dataset, and software/code published by Imperial College London authors between January to September 2020. We sought to understand the distribution of outputs over time by output type, peer review status, publisher, and open access status. Search of Scopus, the institutional repositories, Github, and other databases identified relevant research outputs, which were then combined with Unpaywall open access data and manually‐verified associations between preprints and journal articles. Reports were the earliest output to emerge , but journal articles were the most commonly occurring output type over the entire period . Thirty preprints were identified as connected to a journal article within the set . A total of 52 publishers were identified, of which 4 publishers account for 59.6% of outputs (388/651). The majority of outputs were available open access through gold, hybrid, or green route . The presence of exclusively non‐peer reviewed material from January to March suggests that demand could not be met by journals in this period, and the sector supported this with enhanced preprint services for authors. Connections between preprints and published articles suggests that some authors chose to use both dissemination methods and that, as some publishers also serve across both models, traditional distinctions of output types might be changing. The bronze open access cohort brings widespread ‘free’ access but does not ensure true open access. All outputs published by Imperial College London authors on the topic of COVID‐19 between January and March 2020 were preprints, reports, and software/code and all published without peer review.Across the entire period, January to September 2020, the most common output type from Imperial College London was journal articles, representing 60.8% of all outputs.Widespread open access (OA) compliance was observed, with 66.1% of all outputs available as gold, green, or hybrid OA. A further 30.5% of outputs were granted free reading access at the time of reporting under bronze OA. 3.4% of all outputs are closed access.15.8% of the preprints identified resulted in the publication of a journal article in the same time period, with a median of 60 days between publication of the preprint and the journal article.Fifty‐two publishers were identified across all outputs, of which 4 publishers account for 59.6% of the outputs.et al., https://www.altmetric.com/details/77704842). Many other researchers and groups at Imperial have produced COVID‐19 research in a variety of formats and open access models. We sought to understand the quantity and characteristics of all of Imperial's contributions to COVID‐19 research in order to provide data for the institution to understand its outputs, as well as to provide an institutional cohort perspective to complement the global output level of analysis in other studies on COVID research  in addition to research publications’ , the disease it causes (COVID‐19), and its implications for society have been described as the fastest‐moving production of knowledge in our time  for terms in Title or Keywords and Type as ‘Report’.Datasets collected by search performed on Imperial Research Data Repository for terms in Title and filtered to ‘datasets’. The same search was performed on Zenodo, Google Dataset Search, Open AIRE, and Datacite Search, with these results manually verified to have Imperial authors.Software/code collected by search performed on Imperial College London Github repository, Imperial College London Software Repository, and MRC Centre for Global Infectious Disease Analysis Github repository for terms in Title or About field.Preprints collected by search performed on Imperial College London Current Research Information System (CRIS) for terms in Title, Abstract, or Keywords. Standard Digital Object Identifiers (DOI) prefixes for preprint servers were identified and then used to filter preprint materials in the set.Preprints were also collected through a search on Dimensions for terms in Title and Abstract; ‘Preprint’ in Publication Type and ‘Imperial College London’ in Research Organization. These results were merged with the CRIS preprint results described in Step 5 and deduplicated.All results (excluding arXiv items and software/code without DOIs) were run through the Unpaywall Simple Query Tool to retrieve open access status.Manual update of open access status for arXiv items and software/code.Results were manually screened by exact or similar title and authorship to identify likely preprint/journal article output relationship.For the equivalent of publication date, the earliest found date in the repository referring to the release or any documented action on the output was taken as a proxy publication date. Anonymous authorship practices in software communities introduce uncertainty around author or affiliated institution. Outputs identified from non‐institutionally managed repositories were manually verified to have Imperial authors before inclusion. Multiple versions of the same software/code published in the same repository file were considered as one entity, dated to their earliest found version.Multiple versions of the same preprint that shared a common DOI were counted as a single output, but versions with different DOIs or hosted on different servers or repositories were counted as individual outputs. We could not find a systematic way to identify preprints that also existed as journal articles, so we had to identify these connections manually by similarity of title and author composition. We chose to move the contents of the Unpaywall ‘publisher’ field into the ‘journal name’ field for preprints and inputted manually into the ‘publisher’ field the owner of the server, for example, ‘journal name’ becomes ‘medrXiv’ and ‘publisher’ becomes ‘Cold Spring Harbour Laboratory’.de facto ‘not peer‐reviewed’ for preprints, reports, datasets, and software/code and de facto ‘peer reviewed’ for journal published articles. Exception: Wellcome Open Research and F1000 outputs were considered individually on the basis of their declared peer review status at time of reporting.Peer review status was interpreted as Open access determination was provided by Unpaywall: closed, bronze, green, hybrid, and gold. Closed access is defined as no access to the relevant output without subscription or log in, including membership log ins for a free item. Bronze access is defined as temporary free access via publishers' websites . Green open access is defined as items made available by self‐archiving via institutional or subject repositories or preprint servers (excluding preprints published under a CC BY licence). Hybrid open access is defined as an article distributed under CC BY licence in a subscription journal. Gold open access is defined as CC BY  and for an article, in a journal where all content is distributed fully open access Priem, . We chosA total of 651 outputs were identified from the search. These included journal articles, preprints, software/code, reports, and datasets. See Table Month‐on‐month change in the volume of publication was observed across the period, with some instances of no change: January to February , February to March , March to April , April to May , May to June , June to July , July to August , and August to September   and the final instance of a publication to be Day 257 , we observed the following distribution of publication dates by output type: software/code , reports , preprints , articles , and datasets   and non‐peer reviewed (NPR) revealed January to March outputs were exclusively NPR, but across the entire time period, the majority of the outputs were PR .January ,February ,March ,April ,May ,June ,July ,August ,September  . The median time between the preprint publication and the journal article publication was 60 days (IQR: 25–82.25 days) Fig. .A total of 52 publishers were identified across the outputs. The top 10 publishers by volume of outputs are:Cold Spring Harbour Laboratory (medRxiv and bioRxiv) ;Elsevier ;Wiley ;Springer Nature ;Imperial College London  ;BMJ ;Github ;SAGE ,Cornell University (arXiv)  andOxford University Press .Of these 52 publishers, 4 account for 59.6% of the outputs   of all outputs as open access . The remaining outputs are closed access including bronze . In the months January, February, and March, 100% of outputs were published open access, with the introduction of closed‐access outputs from April onwards. Across the entire period, open access items consistently surpass closed‐access publications in monthly output ;green ;gold ;closed ;bronze  Fig. .Creative Commons licences were observed across journal articles , preprints , software/code , reports , and datasets . Across all output types, the most popular variation of the Creative Commons licence used was CC BY, the least restrictive Creative Commons licence and used by 151 outputs overall . Other open access licences used are broadly permissive: Wiley OpenOnline, Open Government Licence, MIT License, and GNU General Public License , and other existing preprint servers have adapted to prioritize this research or have been established solely for the crisis  was used because, although some databases index preprints , contributor affiliation data associated with preprints is not of sufficient quality or sufficiently widespread to enable comprehensive search with verified affiliation. This is not a fault of the databases but perhaps a dependency on structured and parsable metadata from preprint servers that is not always available. Also, a lack of accessible methods through which to search for connected preprints and published journal articles, also perhaps due to missing associated metadata identifiers; prevents large‐scale or automated data collection and requires associations to be identified manually as in this study. This constraint could be possibly prohibiting the rich insight that could come from easily accessible mapping of preprint and article networks.The presence of 52 publishers found is an indication that authors are served with competitive options from which to choose their own preferred outlet for dissemination and are safeguarded against ‘lock‐in’ from any one provider. Whilst the majority of publishers predominantly serve one output type, e.g. journal publishers to journal articles, some are represented across more than one type – for example, the institutional repository publishing as ‘Imperial College London’ is represented amongst datasets (1), preprints (1), and reports (29). This could be a positive indicator that artificial distinctions in the research life cycle are being replaced with more holistic solutions that offer dissemination for all outputs of research. However, others have raised concern that the representation of commercial publishers across output types poses a threat to equity and value in the research production cycle  types , it is difficult to robustly argue that these outputs were open access as a conscious choice and not a consequence of the NPR output type. There are examples in the full time period of NPR outputs published closed access  in the Imperial Github repository) across the entire time period, but their presence is small . Publisher intervention to convert content to bronze open access is positive but has limitations; the access is not ensured in perpetuity and could be revoked in the future  and the other output types which are not .et al., The green open access share of the data may underrepresent the true number of articles self‐archived, an action that is mandated by the institution's open access policy. This is because outputs would only be classified green when there is no publisher‐hosted option available (Piwowar Authors were served with options to publish rapidly in non‐peer review form and under open access models throughout the entire period, and from January to March, these options were exclusively used. Across the entire period, however, the most commonly observed output was journal articles. The association of some preprints with journal articles suggests that the status of peer review versus non‐peer review is, for some outputs, not binary. This increasing connectedness between the two can also be seen by the presence of publishers serving across both types. That the majority of outputs were published under some form of open access is positive; however, whether the bronze OA cohort is truly compliant with the long‐term needs of this global challenge (World Health Organization, Both authors declare that they are employees of Imperial College London, UK.File S1. Supplementary Information.Click here for additional data file.


Almost 30 years ago, the physicist Paul Ginsparg, at the age of 36 years, installed a server at the Los Alamos National Laboratory to allow the free access to preprints in physics. That can be best described as the foundation of the Open Access movement. It was, however, the increasing financial burden of the “old” subscription model for universities, research institutions and libraries with annual price increases on the one and decreasing financial support on the other hand, which has been a strong argument for Open Access scientific communication, in particular, since the established publishers realized profit margins between 20 and 30% . SubsequBased on these developments, Open Access publications showed a rapid growth between 1993 and 2009, which, since the year 2000, reached an annual average increase of 18% for the number of journals and of 30% for the number of articles. Additionally, in 2009, Open Access articles accounted for almost 8% of all peer-reviewed journal articles . The conhttps://www.eifl.net/) may be lost on the way to the future of publications simply because they cannot afford it. This relates in particular to publishing in a Gold Open Access/Article Processing Charges model. Therefore, there is a need to define opportunities for participation in order to make every participant visible . I. Ihttps: visible . The Eur Surgery . Besides