Publication bias is prevalent within the scientific literature. Whilst there are multiple ideas on how to reduce publication bias, only a minority of journals have made substantive changes to address the problem. We aimed to explore the perceived feasibility of strategies to reduce publication bias by gauging opinions of journal editors (n = 73) and other academics/researchers (n = 160) regarding nine methods of publishing and peer-reviewing research: mandatory publication, negative results journals/articles, open reviewing, peer-review training and accreditation, post-publication review, pre-study publication of methodology, published rejection lists, research registration, and two-stage review. Participants completed a questionnaire asking both quantitative  and qualitative (open-ended) questions regarding the barriers to implementing each suggestion, and their strengths and limitations. Participants were asked to rate the nine suggestions, then choose the method they felt was most effective. Mandatory publication was most popularly selected as the ‘most effective’ method of reducing publication bias for editors (25%), and was the third most popular choice for academics/researchers (14%). The most common selection for academics/researchers was two-stage review (26%), but fewer editors prioritised this (11%). Negative results journals/articles were the second and third most common choices for academics/researchers (21%) and editors (16%), respectively. Editors more commonly chose research registration as ‘most effective’ (21%), which was favoured by only 6% of academics/researchers. Whilst mandatory publication was generally favoured by respondents, it is infeasible to trial at a journal level. Where suggestions have already been implemented , efforts should be made to objectively assess their efficacy. Two-stage review should be further trialled as its popularity amongst academics/researchers suggests it may be well received, though editors may be less receptive. Several underlying barriers to change also emerged, including scientific culture, impact factors, and researcher training; these should be further explored to reduce publication bias. Publication bias is when published research is systematically unrepresentative of all completed studies . The reaBias has the potential to occur at several points of the research process  and no oSome organisations and journals have actively tried to reduce publication bias, whilst several studies have aimed to identify barriers towards fixing the problem as well as solutions e.g. –11). Son. Son11])Whilst studies exploring publication bias provide insight into the current problem and offer potential solutions, to our knowledge none have investigated the perceived feasibility of such solutions by the people that will have to use or implement them. Low perceived feasibility by academics or journal editors may present a barrier to implementing change. In other words, if researchers and/or editors do not want to engage with a new method, it is likely to be ineffective .We therefore aimed to explore different scientific opinions regarding a selection of nine suggested solutions that are currently being used or have been proposed to reduce publication bias, identified from a scoping review of the literature . We alsohttp://www.scimagojr.com/journalrank.php rankings relating to 2014) were contacted. All searches were conducted in July 2015. The editors (n = 157) of 111 journals were contacted. Of these, editors from three journals replied declining to participate. An additional 10 editors from journals that were not directly contacted completed the survey. Impact factors of the journals contacted ranged from 0.12 to 45.22 .In this cross-sectional survey two main groups were recruited, one being journal editors, and the other being academics/researchers. Journal editors included the Editor-in-Chief, Associate and Assistant Editors, and those on Editorial Boards. Editors were recruited via an email to the Editor-in-Chief or similar (e.g. the managing editor) asking them to complete the survey and forward it on to their editorial board. If an editor did not respond to our email, two more reminders were sent, approximately monthly. The journals to contact were chosen via the first 100 articles to come up in a Google Scholar, Jane Biosemantics and PubMed search of ‘publication bias’. Additionally, the top 100 journals according to impact factor (assessed using ‘Academics/researchers’ included those who self-identified as being involved in research (past or present), including those with peer-reviewed published research, postgraduate students, lecturers/professors (with and without research) and full-time researchers. Although the sampling strategy was mainly focused in academia, those working in industry were not excluded.https://www.timeshighereducation.com/world-university-rankings/2015/world-ranking#!/page/0/length/25) to rank respondents from the top 500 universities (those not in the top 500 were ranked ‘500’). All respondents in both surveys were required to give online written informed consent before the survey would allow them to continue.For ‘academics/researchers’, convenience snowball sampling was used. Dissemination was conducted via email and Twitter based on the authors’ own networks. In order to help broaden the scope of the research, some prominent scientists  were also contacted directly to complete and help disseminate the surveys. To gauge representativeness, global university rankings were used : one for editors and one for academics/researchers. The two surveys were virtually identical, with only two key differences. Firstly, some questions were modified in order to suit the respondent, for example “I would consider this for my journal” was replaced with “If a journal implemented this, I would be more inclined to submit to them”, accordingly. Secondly, editors were asked to provide the name of their respective journal, whereas other academics/researchers were required to provide their current/most recent institution. Further demographic questions were added to the survey for other academics/researchers, such as stating their current job role.The surveys were created by three of the study authors  specifically for this research. Whilst the surveys did not undergo any formal validation, the questions chosen were based off previous research investigating similar topics with additional questions added as appropriate in order to tailor the survey to fit our research questions. The nine proposed solutions were based on recent innovations by journals or suggestions made by researchers in the literature (highlighted in The surveys provided suggestions  on how tSD]) or medians (IQR) are reported as appropriate. Chi squared, Kruskal-Wallis or independent samples t-test were used as necessary to test whether there were significant differences in opinions between editors and academics/researchers, or significant differences between early career academics/researchers and established academics/researchers. Data were analysed using SPSS .Quantitative data were visually checked for normality using a histogram, and either means , showing more lower-impact journals responded. A further 160 academics/researchers also responded. Characteristics of the academics/researchers who responded are presented in P = 0.230), but most appropriate content was ranked slightly higher by academics/researchers than editors (P = 0.031).Both editors and academics/researchers were largely in agreement as to why people choose to publish in (their) journals . JournalP = 0.011; P = 0.823) felt that publication bias was a problem in the literature. A small number of editors  felt publication bias was not a problem with qualitative analysis of open-ended responses suggesting some editors did not believe publication bias to be widespread:Most respondents had heard of publication bias (97% of editors and 91% of academics/researchers), though this was lower for early career researchers compared to experienced academics/researchers “P = 0.030) felt the peer-review method of publication is an effective means of publishing quality research, far fewer  felt it was effective at publishing unbiased research .Whilst most respondents . The difference in scores for negative results journals/articles between editors and academics/researchers is driven predominantly by the higher score (p = 0.003) given by academics/researchers with < 10 y experience.Participants were also asked to rate the effectiveness of each suggestion provided “[negative results journals] Accepting negative or null results would require a re-definition of originality and scientific contribution to knowledge” (Ac107)“Future research should therefore qualitatively investigate whether negative results journals/articles do alter perceptions regarding the relative importance of the methodology and results, and quantitatively assess whether this affects publication.p = 0.564) this score was only the fifth highest and was similar between more and less experienced academics/researchers (p = 0.420). Mandatory publication having a comparatively lower score than other suggestions for academics/researchers is possibly due to concerns regarding implementation and enforcement “Research [can] remain unpublished for a variety of legitimate (i.e. non biasing) reasons” (Ac47).It could however also be argued that this would also eliminate industry funding containing limits on publication. Equally, some respondents noted that “p = 0.001), supporting the popularity of academics/researcher choosing it as “most effective” at reducing bias “However, others have noted that this still carries issues, for example the introduction and methods would still be written and submitted after the results are known by the authors, introducing hindsight bias . One aca…statistically significant results have 'news value' and so each decision to publish would become a gamble. There may be a resultant bias towards only accepting 'safe bets' and steering clear of 'long shots'…” (Ac10)““Two-stage review would create even more delays to an already lengthy process” (Ac122), others illustrated that reviewer time commitments and workloads could be reduced particularly when it came to rejecting papers BMC Psychology who have recently announced a pilot study Others were more positive making the point that if there are ways to improve science, they should be tested and implemented as appropriate :All attempts to reduce publication bias should be made regardless of the state of the current system, it should be an evolving process” (Ed71)“Furthermore, reducing publication bias can help the overall scientific process; for example, if more failed experiments are reported, other researchers will not spend time and resources making the same mistakes .Whilst the initial purpose of this survey was intended to gauge opinions of different methods of peer-review and publication in order to reduce publication bias, other related issues emerged. These were primarily the scientific culture, impact factors, and researcher training (discussed below). The purpose of presenting these points is to encourage further research into these areas in terms of how they impact bias and identify options for change, particularly as similar themes have emerged in previous research e.g. ,10)..10]).Many respondents commented on the overall culture of science, with a common theme being that the academic expectations and the perceptions of quality science need changing:Scientists should be employed and promoted on the basis of the quality and integrity of their research, not by counting the number of publications in prestigious journals and the number of citations” (Ed61)“Crooked academic targets and metrics need to go” (Ac42)“A particular issue consistently raised regarding scientific culture was regarding (young) academics on short term contracts. Short term contracts can lead to bias as researchers will aim to submit only the most ‘publishable’ results for peer-review at high impact factor journals in order to be competitive for future work:There is too much pressure on academics to publish… If this pressure was relieved, researchers would have more opportunities to conduct better research and publish better papers. Journals would have more space to publish good research, high quality studies with null results, replications, research plans, and so forth” (Ed56)“Several respondents suggested that academia in itself may not be conducive to change:“One major barrier is the resistance to changes in general…” (Ac38)“…researchers have so much going on, and so little time, it would be very hard to gain support for much change that requires substantial behaviour change/extra tasks” (Ac96)Thus trialling the effectiveness of the more favoured suggestions posited in this (and other) research may increase engagement, particularly as two of the most popular ideas  would likely require minimal to no extra work for the majority of academics/researchers.The publication process was viewed as flawed and going against scientific principles by many respondents . More spToo much emphasis on impact factors, which are in fact fairly meaningless” (Ac111)“A key theme was that impact factors are partially driven by the need for research to be interesting to the readers:…part of the reason journals exist is that readers are interested in content… I don’t think most people would be too interested in reading a journal of methodology” (Ed49)“Whilst not a consensus, one academic/researcher provided an alternative perspective that top journals filtering for exciting results was not a problem in itself, but did lead to other problems:As a reader I'm much more interested to learn about the one-in-a-billion discovery of a flying pig in Nature, than about the gazzilion mundane swine observations across many labs that were necessary in order for someone to stumble across such a nugget. In my view the problem is not so much with (some) journals filtering this way, it is with hiring and funding committees over-emphasizing Nature papers in their decisions. This creates an incentive to have a very lenient threshold for crying 'flying pig' and to be less critical of our own findings” (Ac60)“Whilst peer-review training and accreditation was a fairly well-rated suggestion, it was not perceived to be as effective as other suggestions, which is supported by (limited) research showing minimal to no effect from peer-review training (e.g. ). There Training at postgraduate level on the implications of publication bias to prevent students adopting bad practice” (Ac84)“These sentiments have also been highlighted in previous research ,10 with who offer a free service and thus should not be antagonised” (Ed45). Improved researcher training also has the potential to help prevent other core issues which contribute to bias, such as poor study design:Respondents highlighted several elements of the scientific process which appear to be insufficiently taught formally to students, such as how to write and submit a paper (including reporting guidelines) and peer-reviewing. Teaching such issues to students also addresses some respondents concerns with peer-review training, namely that it would demand additional time from reviewers “…I have not seen any evidence that high quality research is not being published. We are however overrun by poor quality research diluting good science” (Ed13)“Some respondents offered other ideas to help reduce bias , whilst lecturers/professors to teach about publication bias, peer-reviewing and related mattersresearchers to design and conduct high quality research that is reported and published accurately and transparentlyinstitutions to judge scientists on their research quality  and to offer longer term contracts that would accommodate the lower frequency of publications that higher-quality work typically demandseditors to send high-quality studies on for review regardless of the findingsreviewers to focus on methodological and reporting qualityall involved in science (including publishers and the media) to value and prioritise high-quality evidenceThere were some concerns that changing publishing practices does not necessarily impact whether or not research is published—a concern which has also been highlighted by previous research (e.g. ):…there are things peer review cannot help, for example if unfavourable data was removed” (Ed71)“Publishing practices are one important aspect of the research process that can hinder publication. As negative results are less likely to get published, and generally take longer to get published , researcOwing to the potential introduction of unintended consequences (e.g. new biases), journals should aim to trial the more feasible options, both in isolation and in combination with other ideas where appropriate. As many noted that some methods would not work well in specific disciplines, it should be the decision of editors to find what ideas would work in their field. Outcomes of trials of publishing practices would need to include predefined markers of study quality (e.g. the Cochrane risk of bias tool); submission and acceptance rates of studies with favourable and unfavourable results; and a measure of the quality of reviewer comments. This could be conducted by randomly assigning submissions to the usual system or a test system and comparing differences; this may however suffer contamination effects if the same editor has to implement multiple systems with different submission processes within the same journal. Alternatively, a blanket introduction of the new system to the journal and retrospectively comparing the pre-defined outcomes.In the context of this research, we propose that journals further trial two-stage reviewing as this was favoured by academics/researchers (thus is likely to be better received) and has the advantage of being relatively easy to test compared to other favoured suggestions (e.g. mandatory publication). In accordance with previous work, it is important that editors are blinded to the findings too which may create further resistance particularly as editors did not rate two-stage reviewing as highly as academics/researchers. Enforcing any new system was a key concern in both our and others’ research, thus testing methods which are both popular and perceived to be effective may improve self-regulation. Journals that are using other methods  should aim to objectively assess the efficacy of these popular and well-rated methods in reducing publication bias.Similar research is still needed to confirm our findings are representative of both academics/researchers and journal editors, particularly those outside of the UK. Further, our work is limited by the choice of potential solutions proposed; other ideas may have been favoured more than the ones provided. The surveys used were designed to be relatively quick to complete in order to encourage participation. Whilst this led to a relatively high response rate compared to similar research in the field, our findings are comparatively superficial; future work should investigate opinions regarding these issues in more depth, such as qualitative interviews or having a narrower focus. Lastly, whilst understanding opinions is important before wider change, once more extensive research has been conducted on methods to reduce publication bias, perceptions of each method may alter; for example if two-stage reviewing is shown to be successful, editors may view it more favourably, and alternatively if it is found to increase reviewing or publication time, academics/researchers may view it less favourably.This survey explored nine suggestions proposed to reduce publication bias. There was some agreement between editors and academics/researchers that certain suggestions may be more effective than others, however not all would be feasible to trial in a study (e.g. mandatory publication). Negative results articles were considered favourably overall, whilst editors preferred research registration and academics/researchers preferred two-stage review. As negative results journals/articles and research registration are already available/required in some journals, efforts should be made to objectively assess the efficacy of these methods in reducing publication bias. Two-stage review should be further trialled by journals to formally assess its efficacy at reducing publication bias as its popularity amongst academics suggests it may be well received.Other issues surrounding academia and research which relate to publication bias were also highlighted. These included the overall scientific culture, journals and impact factors, and researcher training. Further research should explore these issues with a view to finding feasible solutions to help reduce bias and related issues. We have highlighted specific actions that could be taken by different parties involved in research in order to help reduce the wider issues which affect publication bias.S1 File(PDF)Click here for additional data file.S2 File(PDF)Click here for additional data file.S1 TableReasons given as to why each suggestion (Tables a-i) would be effective at reducing publication bias and the barriers or negatives to implementing this system.(DOCX)Click here for additional data file.S2 Table(DOCX)Click here for additional data file.S3 Table(DOCX)Click here for additional data file.


Nursing Open enters its third year, it is worth recording that this January 2016 issue was published almost a month in advance. Little did we expect this kind of success when I was planning the launch of the journal with Wiley partner Griselda Campbell in a small tea room in a very wet and windy Cork, Ireland during the 2014 INANE conference. As ever, it is much easier to predict the past than to predict the future, but I think it is safe to say that we are now established and recognised as a reputable place to publish open access nursing articles. That is my prediction for the past. I hope and assume that the volume of submissions to Nursing Open will continue to increase in 2016 and beyond; and that is my prediction for the future.As Nursing Open, the online open access publishing landscape is dangerous territory, especially for the unwary traveller. Predatory journals, hijacked websites and a plethora of poor quality publishing have not served to engender confidence. I am pleased that Nursing Open is proving to be a safe haven for those wishing to publish open access.As covered in several of my editorials for Nursing Open. However, it strikes me that it is not an editorial team or a published that makes a journal better. Clearly, we have a part to play in setting standards and ensuring a good service to authors. However, it is authors who make journals better by choosing to publish their work in particular places. It is obvious that more authors are ‘voting with their manuscripts’ and choosing to publish in Nursing Open, and this is what increases our volume and visibility and, ultimately, levers up the quality of what we publish.We continually strive to see how we can make the journal better, more prominent and achieve the various goals we have for Nursing Open is promoted through the use of social media and authors and non‐authors can engage with us here. You can follow us on Twitter (@nursingopen), re‐tweet what you find interesting and also correspond with us there. We have a blog and there you can see my comment on some papers that we promote and it is also a place where you can contribute entries and comments relevant to the journal. I would also welcome occasional guest editorials.Nursing Open then visit our website or email nursingopen@wiley.com.Our first 2 years seem to have passed quickly; I look forward to the year ahead. If you need any information about


Peer review is the cornerstone of scholarly publishing and it is essential that peer reviewers are appointed on the basis of their expertise alone. However, it is difficult to check for any bias in the peer-review process because the identity of peer reviewers generally remains confidential. Here, using public information about the identities of 9000 editors and 43000 reviewers from the Frontiers series of journals, we show that women are underrepresented in the peer-review process, that editors of both genders operate with substantial same-gender preference (homophily), and that the mechanisms of this homophily are gender-dependent. We also show that homophily will persist even if numerical parity between genders is reached, highlighting the need for increased efforts to combat subtler forms of gender bias in scholarly publishing.DOI:http://dx.doi.org/10.7554/eLife.21718.001 Peer review has an important role in improving the quality of research papers. It is the “lifeblood of research in academia […] the social structure that subjects research to the critical assessment of other researchers” . This stwww.frontiersin.org) differ from most journals in that they generally disclose the identities of peer reviewers and associate editors alongside each published article in an attempt to increase the transparency and quality of the publication process  when appointing reviewers. Moreover, our analysis suggests that this homophilic tendency will persist even when men and women are fairly represented in the peer-review process. Our results confirm the need for increased efforts to fight against subtler forms of gender bias in scholarly publishing and not just focus on numerical under-representation alone.is appointing” in the editor-to-reviewer network, and “is editing (reviewing) a manuscript of” in the editor (reviewer)-to-author network). As a whole, the networks had an exponentially fast growth in time, with a large fraction of people participating in a connected component of the graph reaching 90% of the total network size. Furthermore, graph theoretical metrics such as shortest path length, small-world index as well as several other network properties have changed little in the 3-5 last considered years . To test this hypothesis, we took the exact same network of peer-review interactions in the Frontiers journals for given, and randomly permuted gender labels among scientists of a given role 2. This pWe then looked for possible differences over the entire distributions of the number of peer-review tasks and authoring contributions for men and women. These distributions are fat-tailed , indicatThe differences in assignment numbers may reflect behavioral or psychological differences between the groups of male and female scientists — either intrinsic or due to sociocultural context . Neverthlocally reachable female reviewers, i.e. reviewers situated at most five links away  homophilic connections could arise by baseline homophily only, taking into account the editor-specific basin of locally reachable male and female reviewers  and we did so separately for male and female editors  affects potential reviewers' willingness to assess a paper . In partThen, how does the population of scientists contributing to the Frontiers series of journals compare to other scientometric populations? First, we compared our authorship data to that of Second, not much data was available concerning gender bias among reviewers and editors, until very recently. Many previous studies cf.  were selOur work calls for a detailed comparison with another recently published report about peer reviewer assignments in 20 journals of the American Geophysical Union (AGU), based on a slightly smaller sample size compared to ours . This stThe phenomenon of gender homophily in peer-reviewing networks have already been described, but these previous reports have reached ambiguous conclusions. Socrates, in Plato’s Phaedrus, already asserted that: “similarity begets friendship”. Homophily – or “attraction for the similar”, not only limited to the gender attribute – is ubiquitous in social networks. Since the classic studies of Importantly, even a slight homophily can influence and alter the way in which information spreads  and opinWe observed very different patterns of homophily for male and female editors, with a widespread homophily across men, while dominated by very few highly homophilic editors for women. After removal of their contribution, homophily became insignificant cf. . This suOne could speculate that other factors might contribute, like friendship or (perceived) status, competency and reputation. These factors might, in turn, be partly depending on gender, e.g. through implicit biases . MultiplOur finding of strongly homophilic “topology-organizer” female editors is reminiscent of the notion of “femocrat” introduced in political studies, referring to the role played by isolated feminists who, after having managed to integrate inside men-dominated decisional organisms, provide a bridge to the spheres of power for the requests of activists outside of them . Now, whIdeally, all scientific interactions are gender-blind. A scientist's status and the provision of resources to scientists should not be influenced by gender but solely depend on the value of the scientific contributions. Access to the publication systems is a critical determinant of a scientist’s success. Accordingly, reviewers and editors, the gatekeepers of the scientific canon, should be particularly sensitive to base their judgment solely on the merit of scientific work. This merit, however, is difficult to determine and any assessment is necessarily influenced by the assessor's view of the field, including his or her personal position in the network of colleagues and the interactions with them .Inbreeding homophily, an increased affinity between persons with similar attributes, appears to be a sociological, population-level trait of human societies. It is only natural, thus, that we find gender homophily in interactions between editors, reviewers and authors. Nonetheless, this inbreeding homophily is damaging to female scientists, whose work ends up being overlooked, due to unconscious negative bias. The phenomenon of inbreeding homophily is also likely not restricted to the peer review of manuscripts, so it needs to be taken into account for grant evaluation, hiring, or when designing mentoring programs. Importantly, it is likely to persist even when numerical balance between genders is achieved . AltogetInitiatives to remove gender-based inequality can roughly be divided into two different categories. On the one hand, “gender mainstreaming”  promotesIn the field of peer review two more specific strategies are available to reduce bias: blind review and automated editorial management. However, both strategies are of limited acceptance and use. First of all, removing the authors’ names is often not sufficiently blinding. References to the authors’ previous publications or to the approving ethics committee all but spell out the authors. Second, while removal of the authors’ names does indeed blind the reviewers to all irrelevant attributes, it also blinds them to relevant meta-data, such as the scientific experience of the authors, which might be considered as relevant by many reviewers. In an attempt to assist editors of Frontiers journals, keyword-based reviewer suggestions are automatically provided to them but the editors remain free to make their own choices. While these gender-blind automated suggestions could already contribute to an assignment that is less influenced by homophily, an editorial management software is also the ideal platform to routinely direct the editor’s attention to the issue of homophily. It could display statistics similar to our Given how engrained homophily is in our nature, the path towards a gender-blind science will be arduous. Yet, with the joined effort of the scientific community to overcome partisanship and discrimination, a merit-based system with equal opportunities for all scientists might just be within reach. After all, which social enterprise would be more apt to follow ratio over instinct than science?http://www.frontiersin.org/SearchIndexFiles/Index_Articles.aspx, as well as the associated XML file if the HTML code of the article web page contained a corresponding reference. Subsequently, articles' metadata  were extracted from the XML files and the web pages. All gathered personal identity information was deleted after inference of individual genders (see later), resulting thus in a fully anonymized data set. In total, we analyzed 41’100 articles published before January 1st, 2016, covering 142 Frontiers journals from Science, Health, Engineering, and Humanities and Social Sciences. Our parsing routine was able to find information about authors in 41’092 of these articles, about reviewers in 39’788 articles , and about editors in 40’405 articles. The anonymized network data is provided as All article data were exclusively obtained from the publicly available articles web pages from the Frontiers Journal Series (RRID:SCR_007214), which was listed (at the date of last data download in March 2016) on: To recognize and identify people re-occurring in more than one article, every person was assigned a unique identifier number (UID). When a contributor was found to be associated to an official profile identification number in the Frontiers database, then we relied on it, directly translating it into a UID (this happened for 71% of contributors). In the remaining cases, we decided whether a record matched another based on the names and affiliations of people. Specifically, for two names to be matched, we required that the surnames coincided and that each given name of the contributor with less given names needed to have a corresponding match in the other contributor's name . In case both contributors' given names consisted of only initials, we required, in addition, that their affiliations were sufficiently similar. Each UID was assigned a gender based on their associated given names . The extracted given names were compared with an extensive name list, assembled from public web-sources, such as:http://japanese.about.com/library/blgirlsname_[a-z].htm,http://japanese.about.com/library/blboysname_[a-z].htm http://www.top-100-baby-names-search.com/chinese-girl-names.html,http://www.top-100-baby-names-search.com/chinese-boys-names.htmlhttp://www.babynames.org.uk https://www.ssa.gov/oact/babynames/limits.html; retrieved March 17, 2016).US census data (Note that some given names (like Andrea) are in use for both men and women. Gender-ambiguous given names present in the US census database were categorized to the gender to which they were more frequently attributed. When a name appeared as both male and female in one of the other sources, or when different sources did not agree on the gender for a name, we decided not to associate that given name with a gender.We validated the gender assignment procedure by performing a web search for 1053 randomly selected people from our data set, and determining their gender based on a picture or the use of gender-specific pronouns in a biographical text. We were able to find such information for 924 out of the 1053 people (88%). The gender automatically assigned by our algorithm to those identified was correct in 96 % of cases. For comparison, we note that the name-gender algorithm used in Our list thus comprised 66605 female and 43482 male names. In addition to the name list, we manually assigned the non-automatically-identified gender of 643 people with a high number of re-occurrences. In total, we were thereby able to assign gender to 131885, that is 87 % of UIDs. All further analyses were done ignoring the remaining 13% of scientists.We represented the available data in directed networks , in whicAll graph analyses  were perIn graph theory, a connected component is a subgraph in which any two vertices are connected to each other by at least one path, and which is connected to no additional vertices in the full graph. The largest of all the connected components of a graph is called its giant component. One can distinguish between the weak giant component (in which the direction of edges is ignored when building inter-node paths) and the strong giant component (in which the direction is taken into account). All the following graph analyses have been performed on the weak giant component of the networks observed at each time.Transitivity undirected (clustering coefficient) is calculated as the ratio of triangles to connected triangles (triplets) in the graph, considering connections between nodes independent of their direction.Average path length calculates the mean of the geodesic directed path lengths between all pairs of nodes in a connected component. The geodesic path length between a given pair of nodes is the minimum number of links needed to travel between the nodes along connected edges.S is defined in S=γ/λ. γ is the undirected transitivity of the graph divided by k/n, which is an approximation for the undirected transitivity of an Erdös-Rényi random graph with n nodes and average degree (in+out) of k. λ is the ratio of the average shortest path length of the graph to ln(n)/ln(k), which is the average shortest path length of an Erdös-Rényi graph with n nodes and average degree k.Small-worldness Statistical significance was established by comparing a feature of the data to its confidence interval (CI). The graphic notations *, ** and *** denote that this feature lay outside the 95%, 99% and 99.9% CI, respectively. Confidence intervals were calculated by recalculating the given feature 10000 times, after permuting gender labels  at a distance of at most 5 directed edges from the considered editor node i. We counted the overall number Aall of reviewer assignments made by these editors , and neglected those editors for which Aall < 62 . We then determined the number Afemale ≤ Aall of reviewer assignments made toward female nodes. We finally assumed i = Apfemale/ Aall.i=Wi – Ai piH, where iA was the total number of assignment made by each considered editor i.We could then compute the local inbreeding homophily measure iq be the probability a person of the same gender is chosen by an editor i, where iq is calculated exactly as ip in the previous paragraph, i.e. by considering all people at most 5 directed edges away from editor i in the editor-to-reviewer network, counting the number of assignments these people gave to people of the same gender and dividing by the total number of assignments these people made. Next, let ik denote the number of assignments editor i gives to a person of the same gender and in the total number of assignments editor i makes. Assuming editor i chooses the gender of a reviewer at random, the probability that i assigns ik out of the in reviewers to have the same gender follows a binomial distribution binom  and Φhom =i assigns at least ik reviews to a person of the same gender.We used a similar technique to assess the impact of the most homophilic editors on the overall network-homophily in the female editor-to-reviewer and male editor-to-reviewer networks. Let  In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.eLife for consideration as a Feature Article. Your article has been reviewed by three peer reviewers, and the evaluation has been overseen by a Senior/Reviewing Editor. The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.Thank you for submitting your article "Gender bias in scholarly peer review" to Summary:This is an interesting and high quality manuscript on a topic that is important and timely. The manuscript (which is based on a novel data set) merits publication after revision to improve readability and replicability.Essential revisions:eLife requires all major data sets associated with an article to be made freely and widely available , in the most useful formats, and according to the relevant reporting standards. (See below for more details).1) To maintain high standards of research reproducibility, and to promote the reuse of new findings, 2) How representative is the population studied by the authors? It would be interesting to compare the population studied by the authors with other large-scale studies to enhance the generalizeability of their findings. For example, it seems as if the authors could compare 3) The authors should more fully acknowledge the limitations of the data set. For example, the authors mention that the names are publicly disclosed . However, several studies have shown that open peer review significantly affects the willingness of certain populations to conduct reviews . Could the fact that names are disclosed make this less generalizable? The limitations of this should be discussed.4) The analyses only included reviewers who agreed to review . However, women might be more likely to say no to requests to reviews for various reasons . Hence, might their underrepresentation stem from self-selection rather than discrimination?5) It would be interesting to comment on the possible effect of age on gender discrepancies at different levels. Specifically, editors might (on average) be older than reviewers, who in turn might (on average) be older than authors . Is it possible that higher gender discrepancy among editors and reviewers is because women might be less represented in older age groups?6) There is sparse information on author name disambiguation (for those that lacked a UID). More information should be provided on this. The authors state that they cannot guarantee a "completely error-free parsing" – could they run a validation test to give an estimate of the error? And was any validation exercise conducted on the name-gender algorithm? How does this compare with other name-gender algorithms?7) Several seminal books on peer review and articles specifically focused on bias in peer review are omitted here. Furthermore, the article fails to engage with the previous literature in a substantial way. Given that this work is one of the largest peer-review studies to-date, it would be useful for the authors to establish how their work refutes or confirms previous work.8) Regarding the third paragraph in the Results section: please include a figure which shows that the gender gap in various contributions cannot be fully explained by population.9) There are many interesting results that warrant more explanation in the text. At present, some of the most interesting results are buried in the figure captions. It would be great to see the authors embed some of these results, in more detail, in the main text.10) Overall, the differences between observed and chance-level participation of women (for instance in 11) The authors advise that editors should be encouraged to make non-homophilic reviewer selection. But if the goal is to increase gender parity, shouldn't women editors be 'allowed' to be more homophilic, at least until the parity achieved? Might it be better to suggest that all editors should try to recruit more women reviewers? Essential revisions:1) To maintain high standards of research reproducibility, and to promote the reuse of new findings, eLife requires all major data sets associated with an article to be made freely and widely available , in the most useful formats, and according to the relevant reporting standards. (See below for more details).We thank the reviewer for the reminder. We have included the anonymized raw data in the revised submission.2) How representative is the population studied by the authors? It would be interesting to compare the population studied by the authors with other large-scale studies to enhance the generalizeability of their findings. For example, it seems as if the authors could compare We thank the reviewer for these suggestions. We have now augmented the Discussion section with a comparison of the Frontiers population with other scientometric populations  and included two tables  and 2 suWhile our data does not allow inferring the reasons for why women contribute less, some studies indeed suggested that women are less productive. These results are now mentioned and discussed as a potential reason in the revised Discussion . However, for completeness, we also refer to other partially conflicting studies indicating that women may have different, not necessarily lower productivity patterns than men . Furthermore, we stress that the key new findings of our study concern homophily  and that3) The authors should more fully acknowledge the limitations of the data set. For example, the authors mention that the names are publicly disclosed . However, several studies have shown that open peer review significantly affects the willingness of certain populations to conduct reviews . Could the fact that names are disclosed make this less generalizable? The limitations of this should be discussed.We thank the reviewer for pointing this out. We agree that open peer review could potentially have effects other than the intended increase in transparency and quality. These effects are now explicitly discussed in the revised manuscript . Despite extensive search we could not find any study that specifically assesses whether open peer review affects the gender specific acceptance rate of reviewer assignments. Should we have missed a crucial publication, we would be very grateful for a reference.We also now systematically compare our findings to a large number of previous studies which considered smaller sample-sizes but had access to more meta-data, since they were initiated journal-side. Overall, as we comment in the extended Discussion, we find women’s participation rates among editors, reviewers and authors of Frontiers journals to be comparable to previous reports, indicating that the Frontiers community is well representative of a general situation.4) The analyses only included reviewers who agreed to review . However, women might be more likely to say no to requests to reviews for various reasons . Hence, might their underrepresentation stem from self-selection rather than discrimination?The reviewer is correct in suggesting that it is critical to distinguish between self-selection and discrimination. Our data set does not allow inferring the reasons for women’s underrepresentation and therefore self-selection (due to various motivations) could potentially play a role. This possibility, along with others, is now acknowledged and discussed in the revised manuscript . Besides this discussion, we also performed a more detailed analysis of the different prevalence of homophily in the male and female subpopulation. Taken together we feel that the data clearly point to same-gender preference of mostly male editors as a contribution to the under-representation of female scientists across the Frontiers series of journals. This conclusion is particularly strengthened by our new analysis of the different patterns of gender homophily for male and female editors -f.5) It would be interesting to comment on the possible effect of age on gender discrepancies at different levels. Specifically, editors might (on average) be older than reviewers, who in turn might (on average) be older than authors . Is it possible that higher gender discrepancy among editors and reviewers is because women might be less represented in older age groups?Our data set does not include scientists’ ages and we are, therefore, unable to analyze how age affects women’s participation in the peer-review system. However, as the number of female scientists decreases with age and rank, the differences seen in the gender fraction of editors, reviewers and authors could well be influenced by the age distribution in these populations. The revised manuscript now mentions this possibility  and we further discuss our results in light of a recent study with access to age data .6) There is sparse information on author name disambiguation (for those that lacked a UID). More information should be provided on this. The authors state that they cannot guarantee a "completely error-free parsing" – could they run a validation test to give an estimate of the error? And was any validation exercise conducted on the name-gender algorithm? How does this compare with other name-gender algorithms?We thank the reviewer for pointing this out. The gender assignment is a critical step of the analysis and we have now included both a more detailed description of the name-gender algorithm and a validation test in the Materials and methods section (subsections “Collection and parsing of data” and “Determination of gender”). While errors are in principle unavoidable when identifying people based on their names, Newman (2001) estimated their rate to be only few percent. Combined with our reliance on user IDs for the vast majority of people, we expect that errors due to name disambiguation are uncommon.To estimate the error rate in gender assignment, we have validated the genders of a random selection of 924 scientists through a manual web search  and compared our assignment error with the supplemental material of Lariviere et al. (2013). Notably, our algorithm’s error rate is smaller than the reported error rate of the name-gender algorithm of Lariviere et al. (2013). Independent of the error rate, we would like to stress that mis-assigning gender can never increase the homophily that we see. Random mis-assignments have the same effect as a small rate of random permutation of gender labels. This would reduce evidence for inbreeding homophily rather than spuriously inducing it Thus, although we cannot reach a 100% correct gender assignment, the finding of inbreeding homophily is robust against this type of uncertainty.7) Several seminal books on peer review and articles specifically focused on bias in peer review are omitted here. Furthermore, the article fails to engage with the previous literature in a substantial way. Given that this work is one of the largest peer-review studies to-date, it would be useful for the authors to establish how their work refutes or confirms previous work.We thank the reviewer for pointing this out. It is true that our previous version had the style of a short letter, rather than that of an extended article. We now rewrote and substantially extended the Discussion section of the manuscript, including generous paragraphs of comparison with other empirical work, the current state of the art and detailed discussions about why our study can clarify previously ambiguous pictures.Despite these comprehensive additions and expanded reference section  we have failed to include the crucial references the reviewer had in mind; we would be keen to receive more explicit guidance.8) Regarding the third paragraph in the Results section: please include a figure which shows that the gender gap in various contributions cannot be fully explained by population.Every figure in our article clearly indicates comparison with confidence intervals associated to a null hypothesis of no bias effects besides numerical differences between male and female populations. Therefore, our figures already showed the requested information.Indeed, our statistical assessment – random shuffling of gender labels – leaves the network structure and the fraction of females at every level intact. In essence, we ask whether the observed contributions could have occurred randomly, if all actors of different gender behaved identically with respect to the contribution in question.We apologize if this important procedural aspect was not clear enough. We have now better highlighted the concept of the shuffling method cf.  for the 9) There are many interesting results that warrant more explanation in the text. At present, some of the most interesting results are buried in the figure captions. It would be great to see the authors embed some of these results, in more detail, in the main text.We thank the reviewer for this suggestion. Former Figure S3, now We further rewrote and substantially extended the Discussion section of the manuscript to now include a discussion of these findings.10) Overall, the differences between observed and chance-level participation of women  Argue why we are confident, in the light of previous findings, to state that discrimination is a critical reason for the remaining inequity ;B) List direct results of subtler forms of gender bias. Notably, even a slight homophily can influence and alter the way in which information spreads and is damaging to female scientists, whose work is perceived with a negative bias by the majority  and;C) Translate the findings to specific structural disadvantages for women .11) The authors advise that editors should be encouraged to make non-homophilic reviewer selection. But if the goal is to increase gender parity, shouldn't women editors be 'allowed' to be more homophilic, at least until the parity achieved? Might it be better to suggest that all editors should try to recruit more women reviewers?The reviewer is asking an interesting question and we agree: on first glance, it might seem like a prudent policy to support homophily among female editors to reach equity.In response to question 1, we would like to highlight, however, that the goal of scientific policy is not equity per se, but equal opportunity and scientific meritocracy. To this end homophily should be reduced, too, as homophily, even in a situation when parity is reached, prevents equal opportunity. This is now detailed in the revised Discussion in the first two paragraphs of the subsection “Conclusions”.In response to question 2, encouraging editors of both genders to recruit more women might foster the advance of excellent female scientists and advertise that women do good science. We stress however that a broad literature corpus in sociology of work environments have shown that the efficacy of top-down measures aiming at enforcing equal representation is poor if not paired with policies aiming at educating deciders about the importance of fighting gender discrimination .Considering both sides of the medal, it is unclear whether the positive effects (faster route to equity) or negative effects  prevail in the long run. We added a corresponding paragraph to the Discussion (last paragraph) and the subsection “Conclusions” (third paragraph).


Peer review, a cornerstone of academia, promotes rigor and relevance in scientific publishing. As educators are encouraged to adopt a more scholarly approach to medical education, peer review is becoming increasingly important. Junior educators both receive the reviews of their peers, and are also asked to participate as reviewers themselves. As such, it is imperative for junior clinician educators to be well-versed in the art of peer reviewing their colleagues’ work. In this article, our goal was to identify and summarize key papers that may be helpful for faculty members interested in learning more about the peer-review process and how to improve their reviewing skills.The online discussions of the 2016–17 Academic Life in Emergency Medicine (ALiEM) Faculty Incubator program included a robust discussion about peer review, which highlighted a number of papers on that topic. We sought to augment this list with further suggestions by guest experts and by an open call on Twitter for other important papers. Via this process, we created a list of 24 total papers on the topic of peer review. After gathering these papers, our authorship group engaged in a consensus-building process incorporating Delphi methods to identify the papers that best described peer review, and also highlighted important tips for new reviewers.We found and reviewed 24 papers. In our results section, we present our authorship group’s top five most highly rated papers on the topic of peer review. We also summarize these papers with respect to their relevance to junior faculty members and to faculty developers.We present five key papers on peer review that can be used for faculty development for novice writers and reviewers. These papers represent a mix of foundational and explanatory papers that may provide some basis from which junior faculty members might build upon as they both undergo the peer-review process and act as reviewers in turn. Peer review is a key component of academic publishing, and aims to provide rigor and relevance to the publishing process.While peer review draws upon skills that many faculty already have, it does require content and process knowledge that is rarely formally taught to novice reviewers. Quality peer review does not necessarily correlate with traditional markers of experience, such as academic rank, research training, or grant funding.The Faculty Incubator was created by the Academic Life in Emergency Medicine (ALiEM) team to provide early-career educators with a community of practice where they can discuss and debate topics relevant to the 21st century medical educator. To that end, we created a one-month module focused on peer review.This paper is a narrative review that highlights some important literature that may assist junior educators who are seeking to learn more about the peer-review process.In the seventh month of the ALiEM Faculty Incubator (September 2016), we discussed the topic of peer review. We monitored the proceedings of this group of educators from September 1–30, 2016. Our online discussions involved both junior faculty members and faculty mentors. While discussions occurred, we gathered the titles of papers that were cited, shared and recommended within our online discussion forum and compiled these into a list.Academic Emergency Medicine Education & Training and the Journal of Graduate Medical Education; 2) A YouTube Live session with Drs. Ellen Weber & Michael Callaham who are both editors of leading EM journals; and 3) a call for important papers regarding peer review on Twitter. We “tweeted” requests to have participants of the #PeerReview, #FOAMed and #MedEd online communities provide suggestions for important papers on the topic of peer review. This list was then augmented by the following: 1) A YouTube Live session with experts Drs. Jonathan Ilgen & Lalena Yarris, who are both medical educators and editors at Once the augmented list was completed, we then conducted a three-round voting process, similar to our previously described Delphi-inspired method to build consensus around which papers to feature.Our ALiEM Faculty Incubator discussions in combination with expert recommendations and social media calls yielded a total of 24 articles. Our procedure allowed us to create a rank-order listing of all these papers in order of perceived relevance, from the most to the least relevant. The top five papers were expanded upon below. Our ratings of all 24 papers are listed in the Our group determined the following papers to be of highest interest and relevance to novice reviewers and faculty developers. The accompanying commentaries are meant to explain the relevance of these papers to junior faculty members, and also highlight considerations for senior faculty members when using these works for faculty development workshops or sessions.Annals of Behavioral Medicine, describing and providing examples of high-quality reviews for that journal. Although the focus of the process is specific to the behavioral and social-science focus of this journal, the general principles are largely applicable to most academic journals and fields. Specifically, the authors raise awareness of the need for more formal reviewer guidance and attempt to do so by way of this manuscript.Lovejoy and colleagues provide an overview of the peer-review process for The authors begin by discussing the roles of the editors and editorial board for the journal and laying out the responsibilities of each. Of special interest to potential reviewers is the process by which action editors select reviewers, highlighting the importance for new reviewers to only identify actual areas of personal expertise. The majority of the article focuses on the actual process of reading a manuscript and drafting the review, including specific considerations pertaining to each of the separate sections of a manuscript. The authors provide a framework for critically appraising manuscripts by explicitly highlighting the roles of the reviewer in order to 1) offer opinions on the strengths and weaknesses of a manuscript, and 2) provide guidance to authors in how to improve scientific process and communication. To conclude, the authors summarize the “do’s and don’ts” of the peer-review process in addition to providing an annotated example of a high-quality review from a paper published previously within the journal.This paper is relevant to junior faculty who wish to participate in peer review for service, personal professional development, and as a scholarly activity for career advancement. The paper provides an understanding of the peer-review process that is crucial to being able to perform the responsibilities of a reviewer. Although not the focus of the article, a common mistake made by novice peer reviewers is overextending themselves: This may include attempting to review beyond the limits of their actual expertise. A better approach is to select fewer areas of expertise in order to build a portfolio of timely, high-quality reviews, expanding knowledge with progression of one’s career, leading to future review opportunities.The largest area of relevance for junior faculty in this article is found in the step-by-step approach to performing a review. The authors provide a guide for reviewers, starting with accepting or declining an invitation to review and concluding with review submission. Additionally, reviewers should consider reading the articles at least once without marking or making comments, just to assess for readability and understanding.This article then provides a concise yet complete series of considerations for each section of a manuscript, which can help guide the novice reviewer’s thought process and ultimately drafting of the review. Additionally, the article provides two different options for organizing the review, highlighting the necessity to identify major versus minor concerns. Novice reviewers may find this article a useful guide, providing a framework for initial reviews that will likely become more intuitive with experience and time.This paper provides a useful “how-to” resource for faculty developers to prepare academic faculty for peer review. It is a broad and comprehensive overview that provides both step-by-step instructions on the process, and examples to highlight how to apply these instructions to an actual review.Twelve Tips series, this paper provides valuable advice for the more novice peer reviewer. The authors discuss the importance of gauging your qualifications, any significant biases, and available time prior to agreeing to review . They also emphasize the role of the reviewer, not only in critically appraising the article itself, but also in determining how well the submission fits within the journal’s style and mission (Tips 2 and 4). They address the importance of confidentiality and professionalism, highlighting the need to keep critiques constructive and reminding the reviewer that the purpose is to strengthen the paper . The last three tips are, perhaps, the most valuable of all. Tip 10 addresses confidential comments to the editor, clarifying what should be included and the importance of consistency between these recommendations and those shared with the authors. Tip 11 emphasizes the differences between educational and basic scientific research, reminding those reviewing in education journals the differences in approaches and limitations. Finally, tip 12 provides a variety of strategies to improve one’s peer-review skills. While isolated interventions have not significantly influenced peer-review skills,As part of the As junior faculty members become involved in peer review, it is important to keep some core components in mind. This paper provides a concise table highlighting key questions for each component of the submission. The table in this paper can serve as a simple one-page guide for the more novice reviewer to help structure his/her reviews. The subsequent tips emphasize some of the less tangible, but equally important, components of the review process. From a professionalism standpoint, this paper reminds the potential reviewer that s/he should ensure that s/he is adequately qualified and unbiased, and able to provide constructive criticism, rather than simply highlighting faults. The paper also highlights the importance of providing an overview, general recommendations, and assessment of suitability for the journal in addition to the discussion of specific suggestions. Finally, the paper highlights numerous strategies for improving one’s peer-review skills. Examples include attending peer-review workshops, reading papers highlighting strategies for producing high quality reviews, reading other reviewers’ comments from the same paper, asking for feedback from the editor and colleagues, and reflection on one’s experiences.This paper provides a helpful roadmap to guide and orient novice reviewers to the many steps and factors impacting the peer-review process, and many of these are concisely summarized in the Research in Medical Education (RIME) conference was performed. Peer reviewers for RIME come from around the world, and all accepted manuscripts are published in a supplement of Academic Medicine. Each masked submission was evaluated by four or five reviewers who work as medical educators. Anonymous comments and a review form are completed by each reviewer. Eight areas are rated on a five-point scale . The eight areas rated are problem statement and background, research design, sampling, instrumentation and data collection, results, conclusion, writing and importance. Finally, each reviewer is asked to use a four-point  global rating and give additional comments on merits or shortcomings of submission.This study sought to explore the strengths and weaknesses of submissions after analyzing peer-reviewer ratings and comments. A content analysis of the 151 peer-reviewed research manuscripts submitted to the 1997 and 1998 Association of American Medical College- sponsored Interestingly, nearly two fifths of the reviewers recommended rejection without any unsatisfactory ratings on the checklist. The top reason for rejection was inappropriate, incomplete or insufficiently described statistics. This was followed by over-interpretation of results. The top reason for manuscript acceptance was importance, timeliness, relevance, and critical pertinent problem. Both good and bad quality of writing was raised by many reviewers, stressing the importance of well-written manuscripts. Acknowledging limitations rather than ignoring them was also deemed important. As summarized by Bordage, “scientific writing demands both good science and writing good manuscripts.”Bordage highlights important items that junior faculty should consider when taking part in the peer-review process. The ability of a reviewer to determine what is a well-written manuscript and what are appropriate statistics for a study seem most important. This article implies that peer reviewers should have a background in statistics in order to be able to interpret analysis and results as appropriate. A junior peer reviewer based on this study should also be able to critically appraise a research project for its well-performed design, timeliness and novel approach. The ability of the study to provide practical, useful implications should also be considered. The junior reviewer must also be able to provide written comments and feedback to the authors in order to provide guidance in what can be improved.This paper is a great launching point for a discussion on how to improve both peer review and quality of writing with junior scientists. By being aware of the common “fatal flaws” encountered in the field of medical education, it is possible to then pay more attention to these problems when reviewing papers. Faculty developers may want to use the lists generated by this paper to create some easy-to-use handouts for guiding junior faculty members when critically appraising their own work as well. Discussions around each of the most common grounds for rejection and acceptance can be used to scaffold journal club proceedings or internal peer-review processes of research units.Medical Education, discusses the importance of understanding and incorporating reviewer comments, even when the author does not entirely agree with them. The author highlights the importance of the peer-review process for improving a manuscript, emphasizing the value in both well-written, high-caliber reviews, as well as those in which the reviewer is unclear or incorrect in their interpretation. In the latter case, Eva emphasizes that peer reviewers are reading submissions much more carefully than the standard readership and that any confusion should prompt the author to reevaluate the text and address any ambiguity. He subsequently discusses the importance of peer review and provides several strategies for improvement, which include the provision of a guideline for reviewers, deliberate feedback, and maximizing opportunities to review.This editorial, written by the editor-in-chief of After devoting significant time and effort to a publication, junior faculty may become frustrated after receiving reviewer critiques, especially when the reviewer expresses confusion over what appeared so clear to the author. This paper reminds the junior faculty member that reviewer comments are valuable both by emphasizing what may have been missed, as well as those aspects which may be unclear to readers. It is advisable after receiving reviewer comments to set the manuscript aside for several days and return after the emotions have passed and empathize with the reviewer’s comments and perspective. Junior faculty may also benefit by seeking feedback from colleagues and mentors prior to submission. Finally, when serving as a reviewer, junior faculty should review the existing guidelines and seek feedback to ensure that they continually improve their peer-review skills.Medical Education reviewer guidelines (com ) highlighted in this article also provide a useful rubric for the types of general issues that reviewers should consider when conducting a review.This paper highlights several important concepts for faculty teaching peer-review skills to others. In particular, adopting the maxims of “Did I learn anything?” and “What could the authors have done to convince me of the argument they are trying to convey?” frames peer review as an activity rooted in the goal of providing actionable feedback to authors that will help them to improve their work . The Lucey succinctly details advice for peer review in plain language. His 10 tips are concise and capture the essentials of peer reviewing. The first tip is to “be professional,” meaning participate in review because it is a professional obligation as well a means to enhance your own writing. Tips two through four are “be pleasant,” “read the invite” [to review] and follow its specific instructions, and “be helpful.” Don’t only identify shortcomings but offer suggestions to fix identified problems. The fifth tip, “be scientific” emphasizes the reviewer’s essential role. The reviewer contribution is expertise in scientific knowledge (not proofreading). Six, “be timely.” Editors will notice when you stick to deadlines (and don’t). Tip seven, “be realistic” about the reviewing role. The reviewer has an important contribution to make, but not the final say in the ultimate decision regarding publication of the paper. Eight, “be empathetic” in the review and treat others the way that you would want to be treated. Tip nine is “be open” to performing a review even if it is not an area of expertise. Generalists  make significant contributions to the readability and practicality of papers. Finally, tip ten is “be organized.” The review is a communication that requires structure and logical flow. Follow the publisher’s recommended review structure (if available). Specifically, start with an overview, give feedback on the paper structure, quality of data sources, investigation methods, methodology, flow of argument, and validity of conclusions. Comment on the paper style/voice and give specific suggestions for improvement.Peer review is an essential part of an academic career and most junior faculty flounder a bit with the first reviews. This article emphasizes the “big picture” of peer review. It is important because it so clearly and simply states the appropriate responsibility and behavior of an excellent and thoughtful reviewer. It provides an easy-to-follow outline of issues that a reviewer must address when evaluating a paper. The most important emphasis of this paper for junior faculty is the advice to not only find flaws in a paper, but help find solutions. This is an essential skill in thinking critically, evaluating scientific literature and in ultimately developing an academic career.This is a pragmatic article that can be used as a springboard for discussing the role and integration of peer-review responsibilities for those new to the job. Faculty developers will find this a useful guide for reminding faculty members (who may have experienced the slings and arrows of blinded peer review) about how to provide positive and constructive peer reviews. This paper may be a useful prophylaxis against the negative feelings that can emerge between reviewees and reviewers, reminding us to be empathetic, helpful, and kind – rather than unremittingly blunt, mean, or sarcastic.As with our previous papers, we did not design this study to be an exhaustive, systematic search of the literature. We attempted to seek assistance with finding more papers by using expert consultation and an open social media call via Twitter using hashtags #MedEd & #FOAMed, which yielded some important recommended papers. Considering the depth and breadth of our final list, we feel that by using these adjunctive methods we have overcome the limitations of our unstructured collection of papers.We provide a reading list on the topic of peer review that may be beneficial as a primer for junior clinician educators and as a potential reading list for senior faculty members leading faculty development efforts. We hope this paper may serve as a guide for clinician educators who are looking to further the development of their own peer-review skills.


The European Academy of Allergy and Clinical Immunology (EAACI) Junior Members (JM) comprise the largest EAACI section with around 4000 clinicians and scientists under 35 years of age working in the field of allergy and clinical immunology. The Junior Member collaboration with Clinical and Translational Allergy Journal is a mutually beneficial relationship providing Junior Members of EAACI with excellent opportunities to publish their work in the Journal, enhance their visibility in their respective field, and get involved with Journal-related activities and processes. In the future, this collaboration will grow, not only by the consolidation of these activities, but also by the implementation of new initiatives, such as a platform for discussing and/or publishing Junior Members’ dissertations in the Journal. From the CTA perspective, the collaboration presents an opportunity to promote a new generation of allergists with experience of conducting and presenting research, with improved skills in critical review. The European Academy of Allergy and Clinical Immunology (EAACI) Junior Members (JM) comprise the largest EAACI section with around 4000 clinicians and scientists under 35 years of age working in the field of allergy and clinical immunology. EAACI JM benefit from free EAACI membership, reduced registration fees for the EAACI events , free online access to the EAACI Journals and educational resources and the possibility of applying for the EAACI Mentorship Programme, Clinical and Research Fellowships, Scholarships and travel grants. The JM working group (WG), founded in 2001 [The primary aim of Clinical and Translational Allergy is to publish high quality translational research in allergy for clinicians and their patients using an open access platform to allow unrestricted availability and dissemination through social media. A secondary objective is to promote high standards of critical assessment of manuscripts through the review process by EAACI Junior Members.Clinical and Translational Allergy, is the youngest journal in the portfolio of EAACI and provides a platform for the dissemination of allergy research and reviews, EAACI task force reports, position papers and guidelines amongst an international scientific audience. It accepts clinical and translational research articles in the field of allergy and immunology and all its articles are open-access and exclusively web-based, thus ensuring a wide dissemination of its content.Monthly selection of the CTA ‘JM Must Read articles‘; these articles are specifically flagged on the CTA homepage, because they are most relevant to fellows in training based on their educational content. Due to the open access policy of the Journal, all juniors can easily download these articles.Active promotion of CTA in the most popular social media networks; all recent and past, highly accessed articles are posted on EAACI’s Facebook and Twitter account with the hashtag #CTA_Journal.CTA articles are invited to be interviewed by a JM. The purpose of this initiative is to promote the article, give JMs the opportunity to meet senior authors personally and discuss details of their work with them. The first interview [Organization and development of webcasted interviews; leading authors of recently published nterview  is alreaThe JM-CTA collaboration aims at promoting the Journal’s scientific output and increasing its visibility amongst the EAACI Junior and Senior members as well as the general allergy-interested community. In addition, JM can extend their influence and input within the Journal. The juniors that comprise the JM-CTA collaboration team are members of the JM WG and are responsible for:Highlighted the CTA ‘JM Must Read articles’ with the EAACI JM logo on CTA’s homepage.Implemented a Twitter live feed in relation to all tweets carrying the hashtag #CTA_Journal on the Journal’s website ensuring quick information access even to web-users not signed up for Twitter.A limited number of full waivers of the Article Processing charge (APC) will be available to EAACI JM’s to publish their original research in addition to Academy position papers and guidelines.Invited qualified EAACI JM to act as reviewers for the Journal. This not only underlines their recognition in the field but also enhances their critical view on research.Launched two quizzes that were created by the JM of the JM-CTA collaboration team in relation to articles published in CTA , 7. UponReleased an open call to submit hot topic reviews with Juniors as first authors. Four of these articles have been published in CTA within the last year. Saluja et al.  publisheThe CTA Editorial Board has embraced all these initiatives and has enhanced them further. Acknowledging the importance of education for the junior allergy physicians and researchers, the CTA Editorial Board supported the following actions:The Junior Member collaboration with Clinical and Translational Allergy Journal is a mutually beneficial relationship providing Junior Members of EAACI with excellent opportunities to publish their work in the Journal, enhance their visibility in their respective field, and get involved with Journal-related activities and processes. In the future, this collaboration will grow, not only by the consolidation of these activities, but also by the implementation of new initiatives, such as a platform for discussing and/or publishing Junior Members’ dissertations in the Journal. From the CTA perspective, the collaboration presents an opportunity to promote a new generation of allergists with experience of conducting and presenting research, with improved skills in critical review.


BJGP Open, our new open access, online-only journal of primary care research, practice, and policy. BJGP Open will have original, high-quality, peer-reviewed primary care research at its core, complemented by case studies, clinical reports, policy analysis, and the opportunity for discussion and debate.Next month will see the launch of BJGP receives far more high quality original material that it can publish: less than one in four research articles is accepted, pressure on the non-research sections of the Journal is intense, and many of the research articles that we decline are of good scientific quality. The same goes for articles submitted to other sections of the Journal. Primary care everywhere needs to continue to build an evidence base for health care and clinical practice, and to continue to make its political and health policy arguments in all health systems. Peer-reviewed publication is an important means of achieving this. The same is true for education and training in general practice and primary care, and we will welcome original observations and analysis in these fields. BJGP Open provides a new opportunity for primary care researchers, clinicians, and policymakers to publish and read a wider scope of original articles of direct relevance to practice, research, teaching, and healthcare delivery.For some time we have been aware that the BJGP Open, the research section will contain, in addition to original clinical and primary healthcare research, protocols for trials and other planned research studies, and reports of well-conducted educational research. We will also be interested in receiving accounts of novel approaches to delivering primary care services, new directions in primary healthcare policy, and political challenges encountered in primary care, from countries around the globe.We hope that we can develop other, more innovative features in our new Journal. For instance, rather than seeing case reports simply as ways of reporting unusual occurrences or as illustrations of well-known principles, we would like to see them interpreted and used more broadly, to illuminate other aspects of care, such as the doctor–patient relationship, continuity and discontinuity, biopsychosocial dimensions of diagnosis and management, new professional roles in primary care, ethical dilemmas, and health policy issues. We hope that, in future issues of BJGP’s social media channels.This will all be supplemented by a rapid response eLetters facility, enabling readers to debate important topics with authors and with each other, and through the BJGP Open will be an open access only journal, which means that Article Processing Charges will be applied and paid by authors, their institutions, or their research funders. We have set these at £1000 for research articles, and £250 for all other articles in the Journal. These prices, which are substantially lower than most other comparable publications, have been waived and then reduced by 50% for authors of articles accepted in the months leading up to the launch of the Journal, but will be applied in full for articles accepted from 1 March 2017 onwards.Open access publishing is now an established and important part of the medical publication landscape, and BJGP Open will be applying for indexing in PubMed as soon as the first 25 peer-reviewed papers have been published, but — as with all new journals — will need to earn its impact factor. However, as the focus in bibliometrics shifts away from journal impact factors to article-level metrics and other measures of ‘reach’ and impact, we will also focus on the use of alternative metrics — altmetrics — to measure the attention received by individual articles in the journal, and on other measures of their dissemination.BJGP Open will offer new opportunities for primary care publishing, and will develop into an influential global journal of record, debate, and discussion. But, of course, its success will, in the end, depend on you — our readers, authors, and reviewers, who are at the centre of this publishing venture and who I hope will share our enthusiasm for it. BJGP Open will be available in the new year at http://bjgpopen.org, for more information visit http://bjgp.org/bjgpopen.We very much hope that


Scientific irreproducibility is a major issue that has recently increased attention from publishers, authors, funders and other players in the scientific arena.  Published literature suggests that 50-80% of all science performed is irreproducible.  While various solutions to this problem have been proposed, none of them are quick and/or cheap.  Here, we propose one way of reducing scientific irreproducibility by asking authors to revisit their previous publications and provide a commentary after five years. We believe that this measure will alert authors not to over sell their results and will help with better planning and execution of their experiments.  We invite scientific journals to adapt this proposal immediately as a prerequisite for publishing. Hardly a day goes by without a screed against perverse incentives in research. It goes like this: Scientists get better rewards for announcing breakthroughs than for producing solid work. The achievements needed to win grants, jobs, and publications - combined with researchers’ (often noble) ambitions - encourage them to build castles in the air.1.After that comes a plea for large-scale change. One recent proposal would require scientists to complete rigid, time-consuming confirmation studies before publishing a single paperWe propose something that is quicker, cheaper, and simpler: Require researchers to write post-publication reflections five years after their papers appear.In these self-reviews, researchers would assess how their claims held up. They should describe whether an invention or discovery was translated or commercialized, and how (or whether!) others could build on their work. The practice would provide a straightforward, non-stigmatized way to identify errors, misinterpretations, and other roadblocks.For many, these-self-reviews would be a welcome opportunity for clarification, celebration, and even self-promotion. But the main advantage is that self-reviews would encourage scientists to think in advance how they might be wrong.How might this work? Let’s consider the sources of irreproducibility. We put this down to a half-dozen causes: Often several occur together in the same paper! Fraud captures the most attention, but is rare. Self-deception, or bias, occurs aplenty. It is easier to attribute an observation to a hoped-for reason than to imagine trivial causes. Who wants to believe that a test result depends on the brand of test tube or day of the week rather than the earliest detectable sign of disease?2.Then there are unrecognized technical deficiencies; researchers who know how to operate a machine, but lack enough experience to recognize artifacts and infelicities. They enter the wrong parameters or use the wrong pipette tips without realizing that they have rendered their data meaningless. Similarly, big data and data crunchers readily produce false interpretations. In 2007, one crystallographer had to retract five prominent papers after discovering a small computer glitchAll of these problems are exacerbated by fragmented science. Projects are now executed in pieces in various laboratories and results knitted together without anyone knowing exactly what happened at each site, so no one is able to bring sufficient scrutiny to bear.In each of these cases, the problems are clear with hindsight. If post-publication self-review was commonplace, some of these problems would become clear as experiments were being planned and conducted.In our own lab, we have made a habit of reflecting on our papers (though not necessarily with a strict five-year timeframe). Though several papers led to work taken up by biotech companies and other scientists, others proved much less valuable than we had hoped. Bias and technical deficiencies are the most prominent reasons behind our papers that did not ‘succeed.’ That realization has made one of us a better mentor and supervisor over time. It has also led to several publications pointing out flaws in common reagents and lab practices.3. They become more accurate in their thinking and more objective when they evaluate evidence.Work by the psychologists Philip Tetlock and Jessica Lerner suggests that simple steps meant to hold people accountable for their judgment calls actually improves their judgmentad hoc. Researchers get credit for a publication well before enough time has passed for the scientific community to really know whether the paper has made a valuable contribution. No wonder that researchers bent on submitting a paper are obsessed with making the best possible case for its acceptance rather than illustrating its limitations. If researchers are forced to consider how well their paper will stand up five years hence, they will be more careful when doing the work and more critical in their analysis.Accountability in science isReflections in Medicine, in which authors of prominent papers could publish their post-publication thoughts, and contacted about 20 prospective authors, who all ignored or refused the request. We believe some did not want to revisit problematic results.About ten years ago, one of us came up with the idea of a new journal, tentatively titledWith the advent of electronic publishing, it is now possible for journals  to create a space for these five-year reflections and to connect them with the original paper.Self-evaluation, based on strict criteria and instructions, can be revealing even if the authors try to inflate the impact of old work. For example, the boldest claims in a scientific paper should be annotated and addressed directly in authors’ reflections. Researchers could also be asked a series of straightforward yes/no questions about whether the results of a paper have changed clinical or scientific practice.Journals, funders, or research institutions could oblige scientists to write self-reflections. Failing to do so would be a red flag. One can imagine a system in which publications in reference lists or literature databases could be annotated as lacking self-review, and so taken less seriously.With luck, care, and enthusiasm, this simple, inexpensive step would counter perverse incentives. Instead of being stigmatized for correcting a paper, researchers would be stigmatized for failing to do so. Junior scientists would learn by example how to read papers critically and design more-rigorous experiments. The public would learn that a paper is not a definitive statement, but a single contributor to a gradually emerging picture of how nature works.In short, self-reflections could demote scientific papers to their rightful place and turn a vicious cycle into a virtuous one. This review article brings forward a recommendation of mitigating the current issues of reproducibility in academic research by encouraging authors to perform post-publication self-review on their studies in 5 years.  The authors of the review article suggest that the implementation of self-review will allow for academic research to be undertaken with better judgment, as accountability in their work is emphasized. The value of self-review post-publication provides in principle, an excellent platform for authors to reflect on the impact and influence of their research while also providing insight on what led to possible reproducible concerns. The review article appropriately includes limitations of their proposal, such as the adoption rate by the scientific community and implementation into research programs. I believe this is an important point that should be emphasized as the value of self-review is highly dependent on the acceptance of the community. The authors insightfully recommend that self-review be included as terms for publication by "Journals, funders, and research institutions" to promote this process.  In summary, this is a well-written review article introducing self-reflection as a means to identify sources of irreproducibility by the authors of academic research. By encouraging self-review, peers will be able to properly place academic findings into the appropriate place in their knowledge base.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. SYNOPSIS: This overview deals with an enlarging key issue related to the reproducibility of published data in journals of all stripes, ranging from ‘high’ to ‘low’ impact quality. A ‘checkpoint’ process is proposed requiring authors to write a 5-year-post-publication ‘reflection’ on the previously published data. Failure to do so would place a ‘red flag’ beside references to that author’s work in subsequent databases and presumably also in subsequent manuscript reference sections quoting that work. The authors suggest a ‘simple’ yes/no checklist to determine whether the results of a paper have changed clinical or scientific practice. CRITIQUE This contribution is a well-written, thoughtful and concise overview of a deepening thorn in the side of published peer-reviewed data in the literature. The strength of the commentary is that the issue of long-term ‘accountability’ for those who publish the data is raised; and a ‘checkpoint’ solution is well described. The weakness in the overview is that the challenge of implementing the ‘simple questionnaire’ and the alternatives for failure of compliance are not dealt with in the text. For instance, one of the authors would have needed to respond to reflect on over 170 publications over the past 5 years. To determine an accurate answer about the subsequent reproducibility of the 170 findings or about the impact of the findings on clinical or basic science practice would represent a considerable challenge; and the request would be an invitation to side-step accuracy in favour of expediency to get the annoying fly off the desk. This issue is not dealt with in the text. Further, although author accountability is of prime importance, the suggested ‘solution’ would not surface the frustration of those e.g. other laboratories or even new trainees in the authors’ laboratories, who have difficulties repeating the published data. Those ‘negative’ findings, which are of immense importance, almost never see the light of day. The article does not deal with that key issue, which merits attention. SUMMARY AND RECOMMENDATION: This brief, well-written overview surfaces a key issue plaguing the scientific literature and offers a valuable process to enhance author accountability via a 5-year post-publication ‘reflection’ mechanism. That said, the above comments raise two issues that the authors may wish to deal with as they finalize their text. Thus, the possible outcomes of their approach  could be dealt with and alternatives to poor outcomes for their process could be suggested. Further, a process that would enable the ‘reporting’ of lack of reproducibility from others in the field, with an opportunity for the authors to respond with ‘further reflections’ would enhance the process of accountability. As an example, this reviewer has had the opportunity to follow an enzyme isolation procedure exactly as described in a JBC manuscript; and the enzyme activity of the product was essentially as expected. That said, a biochemical sequencing of the enzyme product (NOT done in the previous JBC manuscript) showed that the enzyme was not at all the one claimed to be yielded by the published procedure. How would the ‘reflection’ process deal with that issue? One solution would be for the journal to be alerted as to the lack of reproducibility; and for the journal to insist on accountability from EACH AND EVERY ONE of those who have their names on the originally published article. The authors are encouraged to suggest an expanded ‘accountability’ process that would enhance the ‘reflection’ mechanism suggested and surface the lack of reproducibility by others of findings of published data.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. The long term relevance of our published works is a point for both secret pride and disappointment for many continuing in careers with active engagement in research. This opinion article suggests that compelling researchers to reflectively evaluate their past publications and to produce a published commentary on these works (after 5 years) will improve the reliability of published work. In my opinion, this is an idea that may have reached its time and in some ways is long overdue.  While it remains to be seen if this strategy improves the quality of published work (if implemented), the user friendly and low cost proposal made by these authors at least represents a feasible approach toward improving the quality of publications by causing researchers to think twice about what they submit for publication. And this by knowing that publication involves a long term commitment and accountability to the ideas presented and to be revisited by reflective review 5 years later. Certainly, this is a well written and interesting concept that in my opinion merits consideration by journal editors and publishers, and the support of researchers committed to research excellence.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Scientific irreproducibility is indeed a serious problem nowadays. In the current opinion article, the authors state the reason that govern irreproducibility and for the first time they provide a potential method to treat such results. Importantly, their suggested method is based on a self-evaluation by the authors of a published article after a 5-year period. Indeed, this “self-review” process is simple, quick and with no additional cost. The article is very well-written for a broad audience.I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.