Progressive Culture | Scholars & Rogues
Arts & literature, pop culture, media, photography, sports and climate
The New Constitution
Climate Illogic
The Tournament of Rock
The Ramsey Case
Harvey Pekar Tribute
S&R LitJournal
View scholarsandrogues's profile on Facebook
View scholars_rogues's profile on Twitter
The 2019 #Asshole of the Year is #MitchMcConnell scholarsandrogues.com/2019/12/31/the… #AotY #gop #MoscowMitch https://t.co/nP7TS5SrKn 2 weeks ago
scholarsandrogues.com/2019/12/30/201… 2 weeks ago
More guns isn't the answer to the US' gun problem scholarsandrogues.com/2019/12/30/mor… https://t.co/VJxOy55yq9 2 weeks ago
RT @docslammy: @RideRTD: is this how you treat #disabled passengers? scholarsandrogues.com/2019/12/06/den… via @wordpressdotcom https://t.co/a4gNdJEo5b 1 month ago
The final word on @Kaepernick7 (for now) scholarsandrogues.com/2019/11/24/the… @nfl @stephenasmith #jayz #racism #plantation… twitter.com/i/web/status/1… 1 month ago
Follow @scholars_rogues
Why scientific peer review matters
By Brian Angliss on April 5, 2009 • ( 11 Comments )
"Peer review isn't all it's cracked up to be," and that phrase's many variants, is something you year a lot these days. It's a complaint that is heard most often with reference to climate science, and most often by people who understand neither the science behind climate disruption nor the purpose of peer review. So, as someone who's undergone peer review repeatedly in both academia and as an electrical engineer, I'd like to explain why peer review matters.
First, a little explanation of what peer review is.
Peer review is a formal process instituted by an organization to ensure that a quality product is produced. In the case of scientific peer review, the product is scientific papers that meet a presumably high standard of scientific accuracy and professionalism. Scientific peer review involves distributing review copies of submitted papers to experts in the field in advance of publication. These usually anonymous experts read and offer comments that suggest improvements in the style, data, or thought processes described in the paper. The comments are then given to the paper's authors who incorporate the suggestion, resubmit the paper to the journal for another round of review and, if there are no additional necessary improvements, the paper is then published.
In my own profession of electrical engineering, the peer reviews tend to be more in-depth than what I just described. Here's the usual list of reviews. I'll pick up after the specification phase, when engineers get handed a design and told "design this." Any one of the reviews described below may be done repeatedly if the first review finds a significant number of problems.
Project internal design review: When the design is largely complete, the entire design team sits down and tries to ensure that nothing major was missed. Problems are identified and corrected before the next review
Independent design review: other engineers in the company, often senior engineers with decades of experience, are brought in to review the design with the engineer in charge and other members of the team. Problems are identified and corrected before the next review.
Customer design review (optional): If you're doing contract engineering work, then the customer often wants to be involved in the design review process. This review may include any outside experts that the customer hires to review the work as well. This review always happens after the independent review.
Design document review: In the process of writing up the documentation in enough detail that another engineer could replicate the design if the engineer in charge were struck by lightning, small errors can be discovered. These errors would be corrected and the documentation review (usually by independent reviewers again) often turns up further small errors missed in earlier reviews.
Preliminary layout review: Just because you can design an electronic circuit doesn't mean it can be built. Design problems that cause manufacturing difficulties are first identified here.
Final layout review: Due to (usually) minor changes that are made during layout, the schematic is reviewed yet again at the same time that the layout of the printed circuit board is.
One major misconception about all varieties of peer review is that the reviews guarantee no errors in the final product. Not even five or more engineering design reviews described above can ensure that there aren't any errors in the design. In fact, one of the differences between a designer with experience and one fresh out of college is that the experienced designer plans for the inevitable errors that are discovered in testing, while the inexperienced designer thinks his or her design won't have errors because it was reviewed.
While catching errors is an important purpose of peer review, what kind of errors are caught and corrected depends greatly on who the expert reviewers are. If a climatology paper that relies on detailed knowledge of statistics has an error in the statistics, sending the paper to only expert climatology reviewers might not turn up the statistical error. Similarly, if there's a power problem in an electronics design, it might only be detected if a power supply engineer was one of the reviewers.
Ultimately, though, this isn't a problem – not with peer review in general and not with scientific publications in particular. Just as there's a check on an engineering design review called "product testing," the publication of a peer reviewed paper provides the final check as well. Once a scientific paper is published, it's reasonably likely that someone reading the paper will discover any errors. Discovered errors are corrected either by a retraction if the error is significant, or by the submission of another another paper describing the errors that also goes through the peer review and publication cycle.
So why does scientific peer review matter? Because the peer review process described above is a key component in a process that continually improves the overall quality of scientific research. And because participating in that process is the price of entry to be taken seriously as a scientist.
It takes confidence in your skills and knowledge in order to put yourself through the wringer that is publication. After all, your discoveries and data are then publicly available for all to critique, agree with, or even mock. The process alone provides a level of confidence in the scientific accuracy and veracity of the papers that are published.
After all, anyone can publish a blog filled with with so many numbers that it looks legitimate, but only a scientist would subject him or herself to peer review.
As a final aside, there is no such thing as a "peer reviewed book." Books are published not based on the quality of the science as determined by anonymous expert reviewers, but rather on a publisher's and editor's determination of how well the book will sell. If the science happens to be interesting and accurate, great, but at best that's a secondary consideration over profit. Scientific journals publish papers based on the quality of the science first and foremost, and then charge whatever the market will bear. Given that a single journal can cost over $1,000 per year for a subscription, the market is willing to bear a great deal.
Click to share on Fark (Opens in new window)
Categories: American Culture, Media/Entertainment, Science/Technology
Tagged as: American culture, electrical engineering, Icecap, peer review
Michael Vick's second crime almost as bad as his first
World's longest-running war for independence — or exercise in futility? (Pt. 3)
JS OBrien says:
Thanks Brian. Great background on how engineers do it.
One thing I've heard about both peer review and publication from scientists is that it's not only rigorous, but can get rancorous (if collegially so). I've heard it described as more of an acid test than a stress test, especially if the paper is at all controversial.
I think what John Q. Public often doesn't understand is that science is a kind of jungle of ideas in which only the best ideas survive.
At least, I've heard it explained to me that way.
Ubertramp says:
This is how it is ideally supposed to work, sure. But I guarantee you that politics gets involved in the review process. Not only in publishing manuscripts, but also in grant reviews. Stuff like this happens more often than you might think:
I'm not saying that there's a better way out there. I've published 30+ manuscripts through this process and I can attest to the amount of pain and frustration it causes. But hell if I can figure out a better way to do it.
Brian Angliss says:
I'm not trying to pretend the process is perfect. Politics certainly plays a part, but because the peer review system is relatively rough and tumble, it's also self-correcting over the long run. Like you, though, I don't know of any better way to do it at this point.
Funny you should choose to write about this now. This came out a few days ago.
http://www.nsf.gov/news/news_summ.jsp?cntn_id=114497&govDel=USNSF_51
It's kind of a cheer leading session for the NSF process, but still a tad bit interesting.
Djerrid says:
To paraphrase Churchill: Peer review is the worst method of accumulating scientific knowledge except for all others that have been tried.
Thanks for the link Ubertramp. I considered doing something similar in my anthropology days but never ha access. I might have to pick that book up.
And thanks for the discussion Brian. When I was looking for a job a few years ago, I just looked for something in academic research, period. I've been fascinated by the methodology ever since I was first introduced to it and never looked back.
Djerrid, I almost used that Churchill quote about democracy. 🙂
Pingback: Peer Review and the Science Versus Opinion Smackdown | FollowGreen.com
johns says:
As a scientist, I'd agree with the comments above regarding some contention in the review process, but there are two big differences in science, including the social sciences.
First, there is a very wide range of rigor brought to bear in the review process and how much depends on the specific journal and the professional society involved. Even the superficial procedures differ widely. For example, Science magazine, of the AAAS, used to have only two reviewers, wherein the disapproval of either could scotch publication without appeal. Others have 3 or more reviewers with feedback to the authors simiilar to that described by Brian. Within those superficial differences are varying standards of rigor with regard to relevance, originality etc.
Second, and this is almost uniformly misunderstood by the public, successful peer review and publication in science is only the beginning of the "evaluation" process – not certification that other scientists in the field, or even a majority of them,.agree with theory or models proffered or the authors' conclusions, given the methods and procedures used. Review and publication only means that a study meets minimum standards of design, execution, interpretation, and reporting for the society involved. If the reviewers overlooked uncontrolled variables that other theorists and "bench" or "field" researchers feel are relevant, this will only come out in subsequent letters or other publications. If the theory and hypothesis tested fall short, in the opinions of others, those disagreements will be brought out only in subsequent publications. There are fundamental differences in the way engineers and scientists approach research, discovery and acceptance with those differences being fully reflected in their respective review, publication and evaluation processes.
Unfortunately, the second factor leads to endless "false alarms" by "science writers" and journalists (but, for which "science" is often blamed) who take the contents and conclusions of each peer-reviewed and published paper as "the truth" at face value without understand the context of all other historical and current research in the field. Typically, this is evidenced by successive contradictory announcements, each touted as a new breakthrough but contradicting the previous one, when, in fact, it is only their – and sometimes the authors' – selective view of the field with respect to their favored theory (their chosen device for organizing the known facts). In the worst case, the "finding" is simply ignored by all others. While this may appear untidy, indeterminate and frustrating to some, it is how science "stays on track" and, eventually, comes up with sustainable explanations for nature that are widely accepted.
johns, I agree. One paper isn't very strong by itself. As scientists, we kind of intuitively recognize that it's the network of related pubs that makes the stronger argument. Hell, that's half the reason we HAVE a bibliography section at the end of every paper. However, I've also noticed that some journals don't like to publish data if it looks too much like a repeat of other work. To me, it always made sense to have at least a few different labs try to run the same experiment.
For this very reason, I'm also a bit concerned with the way the recent NIH RC1 Stimulus package/Challenge grants are being written. Not only are we supposed to skip the intro, background, and preliminary data sections, but we are limited to exactly 1 page of references for a 12 page proposal. I'm not sure how that's going to work out. Either the reviewers have to be experts in every aspect of every grant proposal, or a lot of stuff is going to slip through the cracks that shouldn't. I understand that this is supposed to help the reviewers get through everything they have to get through in the review process, but it seems like a dangerous way to go about it.
Ubertramp, I think the favor by journals of exact replication varies all over the map with the theoretical importance of the finding, the developmental stage of the science (maturity of the theory), and the cost of replication. "Big Science" projects occur in well developed fields where theory is mature (or, at least, successfully represented as such – I'm getting old waiting for the oft-promised grand unified theory of everything) and, together with cost, make the suggestion of replication almost foolish. In the biological, and the least mature social sciences, exact replication is usually treated as a "note" or "communication" (To replicate is "no big deal" and to fail brings suspicion on your methods, all of which can make killing the result of shabby research very difficult.). Alternatively, my preference and easily published, the attempted replication may be embedded with other experimental variables or controls of a larger experiment that aims, if the replication succeeds, to add to or limit the scope of the original conclusion and, if the replication fails, guarantee an unambiguous alternative explanation. Not always easy, but that's the challenge of it.
Re NIH. Some years back, when last I dealt with it, the competition for grants had reached an insane level wherein many really wonderful proposals were being routinely rejected for lack of funds. The differences between the accepted and rejected were imperceptible. You had to be quite exceptional to even enter what was, in effect, a lottery. I'm not trying to discourage you, only show my admiration for your ambition and talent – obvious, by your even being in that race.
Your concern with the "summary proposal" method is, likely, well founded. One can only hope for competent reviewers that, also, will pool their talents when in doubt and not, as you fear, hand out moneys for outlandish promises or sequester too much of it in their own NIH labs – which has been a problem in the past. However, with a pot of added money, the odds are certainly improved and, hopefully, a large number of worthy researchers will be funded. Good luck to you.
Competition is still pretty insane and it's still very much a crapshoot. Haha! But I keep plugging away. I don't think they'll fund anything "outlandish" (which isn't necessarily a good thing), but I do think that this method of reviewing grant proposals will limit creativity. My guess is that the grant reviewers will give high marks to ideas they are familiar with, and trash anything that might be on the edge. But we'll see.
Leave us a reply. All replies are moderated according to our Comment Policy (see "About S&R") Cancel reply
S&R on Facebook
Does the Culture Index accurately describe me? You make the call.
Your Daily Devotional for January 17, 2020
There's an alternative to Facebook: 8 steps to make MeWe work
Dante's Inferno and contrapasso
Anti-global heating claims - a reasonably thorough debunking
Time to kiss off online dating: a long-overdue farewell to Match.com
The billion billion stones of Jasper Beach
The 2019 Asshole of the Year is Mitch McConnell
Review: MeWe is superior to Facebook in virtually every way
Mark Zuckerberg, Mitch McConnell and Fake Christians named Donald Trump Assholes of the Year for 2018
Dead Mule School of Southern Literature
Rude Pundit
Brisbane Graphic Arts Museum (Dan Ryan)
Smirking Chimp
From Pizza to Politics
5280 Lens Mafia
New Southern Gentleman
Categories Select Category American Culture Anime Binge Arts/Literature ArtSunday ArtsWeek Asshole of the Year Business/Finance Crime/Corruption Daily Devotional Economy Education Energy Environment/Nature Family/Marriage Featured Features Food/Drink Freedom/Privacy Funny Generations Health History HobbyWeek Infrastructure Internet/Telecom/Social Media Journalism l'etranger Leisure/Travel Lex Veritas LGBT Media/Entertainment Music/Popular Culture My Week of Shooting Nota Bene Personal Narrative Photography podcasts Politics/Law/Government Quotabull Race/Gender Religion & Philosophy ReligionWeek Renewable Journal S&R Fiction S&R Literature S&R Nonfiction S&R Poetry Saturday Video Roundup SchadenFriday scholars and rogues Scholarship/Theory Science/Technology Scroguely Works Scrogues Converse Scrogues Gallery Scrogues Nightstand Sex slider SNRGTR Sports The Couch Podcast Tournament of Rock TunesDay TunesWeek Uncategorized United States War/Security Week in Snark Weekly Carboholic What's It Wednesday Words Matter WordsDay World
Archives Select Month January 2020 (17) December 2019 (36) November 2019 (31) October 2019 (28) September 2019 (32) August 2019 (39) July 2019 (33) June 2019 (28) May 2019 (36) April 2019 (40) March 2019 (46) February 2019 (30) January 2019 (45) December 2018 (39) November 2018 (36) October 2018 (41) September 2018 (36) August 2018 (45) July 2018 (30) June 2018 (52) May 2018 (50) April 2018 (50) March 2018 (58) February 2018 (68) January 2018 (57) December 2017 (53) November 2017 (47) October 2017 (63) September 2017 (77) August 2017 (75) July 2017 (60) June 2017 (69) May 2017 (68) April 2017 (76) March 2017 (69) February 2017 (63) January 2017 (79) December 2016 (69) November 2016 (67) October 2016 (57) September 2016 (53) August 2016 (60) July 2016 (62) June 2016 (65) May 2016 (61) April 2016 (45) March 2016 (51) February 2016 (58) January 2016 (52) December 2015 (63) November 2015 (58) October 2015 (54) September 2015 (55) August 2015 (60) July 2015 (60) June 2015 (63) May 2015 (71) April 2015 (72) March 2015 (82) February 2015 (78) January 2015 (58) December 2014 (68) November 2014 (61) October 2014 (63) September 2014 (64) August 2014 (78) July 2014 (78) June 2014 (77) May 2014 (69) April 2014 (69) March 2014 (69) February 2014 (86) January 2014 (114) December 2013 (96) November 2013 (70) October 2013 (64) September 2013 (45) August 2013 (59) July 2013 (78) June 2013 (58) May 2013 (66) April 2013 (57) March 2013 (57) February 2013 (68) January 2013 (85) December 2012 (72) November 2012 (75) October 2012 (64) September 2012 (75) August 2012 (89) July 2012 (90) June 2012 (90) May 2012 (103) April 2012 (79) March 2012 (101) February 2012 (98) January 2012 (85) December 2011 (89) November 2011 (80) October 2011 (80) September 2011 (83) August 2011 (61) July 2011 (88) June 2011 (90) May 2011 (88) April 2011 (81) March 2011 (86) February 2011 (59) January 2011 (70) December 2010 (57) November 2010 (68) October 2010 (57) September 2010 (59) August 2010 (63) July 2010 (66) June 2010 (47) May 2010 (61) April 2010 (66) March 2010 (54) February 2010 (54) January 2010 (66) December 2009 (85) November 2009 (80) October 2009 (107) September 2009 (86) August 2009 (75) July 2009 (64) June 2009 (71) May 2009 (71) April 2009 (80) March 2009 (70) February 2009 (55) January 2009 (89) December 2008 (64) November 2008 (82) October 2008 (92) September 2008 (119) August 2008 (137) July 2008 (81) June 2008 (93) May 2008 (76) April 2008 (86) March 2008 (124) February 2008 (106) January 2008 (108) December 2007 (121) November 2007 (86) October 2007 (70) September 2007 (102) August 2007 (104) July 2007 (120) June 2007 (134) May 2007 (138) April 2007 (90)
Top categories: Politics/Law/Government government