Measure of review bias I was reading the other day this blog post, about &quot;how to peer review&quot;, and one passage struck me:

Don't review like a grad student
Reviews written by graduate students are among the most negative.
Grad students are often on the receiving end of negative reviews, because they are just learning how to write papers.

So, even though I'm no longer a grad student, I sometimes have the feeling that I tend to be a more negative reviewer than others, but it's actually hard to measure (maybe I'm just reviewing mostly bad papers?!), since I don't always have access to the other reviews of the papers I'm reviewing (due to the fact that I've been more often acting as an external reviewer than as a PC member).
I believe that the quality of my writing has improved with the feedback I've received from the reviewers of my paper, but because I don't have any feedback on my reviews, it's hard to know if and how I can improve them. So my question is: is there a way to measure my own reviewing bias?
The first thing that would come to my mind would be a set of papers reviewed by many other reviewers, and a result like: &quot;you're in the x% more negative reviewers&quot;. Of course, I'm not implying that I would be automatically less severe when reviewing, but sometimes, in case of doubt, it's good to know one's own bias.
There aren’t many ways for your to evaluate how your reviews stand against others, but there is at least one. You can keep a tab of the papers you have reviewed, and look for them a year or so after the review. See if your reviews correlate with the publication (or lack thereof) of the papers in the respective journals… if papers you have suggested to reject are frequently accepted by the editor, then you give typically more negative reviews than other reviewers of your community.

Of course, this is a very black-and-white picture (rejection/publication). But, I don't think you can actually do much better, because “negativity” can hardly be measured quantitatively.



PS: I wouldn't call it a reviewing bias. A review is the editor asking you your frankly assessment of a publication. Toning it up or down because you fell that you do not conform to the “mean” of your field would probably diminish your value as a reviewer. If the editor does not think your reviews are helpful to him, he’ll just stop sending you paper to review! 

I'll finish with an anecdote: there's a journal for which I have recently given a series of quite negative reviews (well documented, and justified in my eye), and I have actually received some positive feedback about my reviews from the editor.