NSF Panelist Reviewing Rubric/Criteria I will be serving in my very first NSF Panel as a panelist and reviewer. While I am excited, I am also anxious to do a good job... 
Please provide any words of wisdom to be successful in this endeavor. Also please suggest if there are any Rubric that you aware with respect to 
reviewing NSF proposals... If so could you share / provide pointers to it...
I don't know much about the NSF process, but can share my thoughts on the NIH process, and maybe others can comment on how this applies.  I think some of this is directly applicable.

Remember, you're reviewing grants and not papers.  You have reached the state of your career where your actions can DIRECTLY IMPACT somebody else's career, and fairly immediately.  It could conceivably even end some careers.  Do not take this responsibility lightly.

The biggest issue that I see come up is "New Reviewer Syndrome", where a reviewer feels compelled to make him or herself heard in a room full of people, even though the criticisms offered are nickel and dime stuff.  You can't offer a list of five trivial matters and then recommend a high priority score.  It doesn't work that way.  If you don't have substantive criticism, best to keep quiet.  

That said, if you have substantive criticism that you believe can turn the grant from unfunded into one that's funded on the next round, SAY IT AND SAY IT LOUDLY.  Communicate it in a way that the applicant can hear it and understand it, and know that this is the issue separating him or her from a recommendation for funding.  Don't make the applicant guess about how important you consider this issue.  Failure of the applicant to understand what the study section wants will result in extra rounds, lost time, negative career impact, etc.  If the person running the section sends you some synopsis of the review, read it and make sure that it conveys the feeling in the room during discussion.  If you are asked to write such a synopsis, take that job VERY SERIOUSLY.

Another issue I have seen repeatedly is that its very tempting to score the grant that you wish the applicant wrote, and not the grant the applicant wrote.  Don't judge the grant based on how you would do that research, and remember that there's more than one way to skin a cat.  

Your institute will provide you with some good guidelines, but here's a framework to start with:


What question is the applicant asking of the proposed study?
Is that question worth getting an answer to?
Will the proposed study answer that question?
Is the applicant/applicant's environment capable of successfully carrying out that research?


When you're outside of the review room, KEEP YOUR MOUTH SHUT.  Never violate the confidentiality of the process.  I've had people come up to me in conferences and start telling me about what was in my applications, even though they were not on the study section.  This put me in a situation where I had to moan to my program officer that they had a leak, and they should shut it.

Listen to your peers in the room.  Some have been doing this a lot longer than you, and some are in a better position to judge a particular application than you are.  Decide if their input sways your opinion, and act accordingly.

Don't get provincial about defending your area of research to the study section.  Your colleagues in the room will be doing enough of this. Research that is what you do isn't by definition good research, and research that is not what you do is not by definition bad research.

Lastly, whatever serves at NSF as the equivalent of an NIH review officer is there to help you.  Read and understand whatever review guidelines you are sent by the powers that be.  At NIH, many instructions go toward trying to get the study sections to use the full range of scores.  If you can do this, this makes life much easier for the review group.  If you can't, you really start getting the feeling that the pay cutoff is quite arbitrary.