Monday, March 19, 2012

Proposal Excellenzzzzzzzzz

When I am asked to review a grant proposal by a program director at NSF (doesn't matter which program), I almost always say yes. There have been a few cases that I can remember when I was sent a proposal quite far out of my field of expertise, and I said no to those, but it is rare that I am asked to review proposals that I have no reasonable basis to judge. This is not because I am so awesomely expert in so many things but because the program directors do an excellent job of targeting reviewers.

Of course, saying yes to review a proposal on a topic that you find somewhat-to-very interesting doesn't mean the proposal will be good (or very good). And, unless you have reviewed proposals and/or manuscripts by the PI/coPIs before, you may not know in advance whether it is likely to be well written or not.

So it can be hard to predict what the review-experience is going to be like, even if you have some basic information at the time of agreeing to the review. I am thinking in particular of a proposal that I reviewed not long ago. The proposal was on an interesting topic, the writing was mostly good in a technical sense, and the proposed work involved what seemed to be the appropriate approach (methods etc.), and yet.. it was extremely difficult for me to read more than a page without flinging it aside (or, in reality, the electronic equivalent of flinging, as I was reading it as a pdf).
Why? Because it was so so so so boring to read this proposal. It was very painful.

Is my attention span getting shorter as I get older and busier and more jaded? Do I need lots of insightful illustrations and formatted text telling me what is important? Do I have to be grabbed in the project summary and/or first paragraph of the project description or else I have trouble being interested in the proposal? Answers at the end of this post.

Eventually I did read the proposal all the way through, of course, and I even re-read what I thought were some critical parts, but it was a slog from start to finish. I had to work really hard to figure out that the proposed work was of some interest and significance. The PI did not highlight the significance, but it was in there, somewhere.
I have written about related issues before: How hard do/should we as reviewers work to see the good (or excellence) in a proposal that has some technical flaws? In many cases, the flaws are in the writing, but in this case it wasn't so much how the sentences were constructed (each one by itself was fine), but in how the proposal as a whole was framed. If I had to work so hard to extract the interesting essence of the proposal, does that mean I will/should rate the proposal lower than I would if the proposal were technically excellent (in addition to containing interesting proposed work)?

(I am ignoring, in this particular discussion, the issue of giving early-career people a bit of a break; i.e., not expecting proposal perfection for those who haven't written many or any before.)

I fear the answer is yes, in a way. There was no way I was giving this proposal a rating of Excellent (the highest rating for an NSF proposal). If the proposal itself had been really well put together, showing more vision in general about the work and how to explain it, it could possibly have been in the Excellent range, but given what I saw in the proposal: no. Ignoring the fact that some of the shortcomings in understanding the proposal may have been my own, can I assume that the dense, complicated, non-linear, dry presentation of the proposed work is an indication of how the research itself will be undertaken, understood, and communicated? I decided the answer to that was also yes, and hence my overall positive but not-Excellent review.

That might sound harsh, so let me say that my first impression of the proposal was even more negative, but my opinion improved with more careful reading and thought. The proposed work is good, even very good. In the end, my rating was quite positive, and (I hope) backed up by what I wrote in my review. So:

Is my attention span getting shorter as I get older and busier and more jaded?Yes

Do I need lots of insightful illustrations and formatted text
telling me what is important?Maybe -- it would be nice, anyway. Of course, writing "The hypothesis to be tested is ..." in bold italics underlined and set off with an indented bullet, surrounded by spaces and maybe some nice subtle shading and prominent on page 1 of a proposal doesn't guarantee you won't get a reviewer who writes "It is not clear to me what hypothesis is being tested", but one can reduce the chances of getting comments like this.

Do I have to be grabbed in the project
summary and/or first paragraph of the project description or else I have
trouble being interested in the proposal?Not this time, but I think in general, yes. I eventually forced myself to sit down and read the whole proposal carefully, painful though it was to do so, and saw some interesting things buried in the dense text, but I wouldn't want to risk this myself when writing a proposal. Perhaps my review would have been more positive if I hadn't found it a huge chore to read the proposal. An annoyed, weary, bored reviewer is probably best avoided.

17 comments:

SocSciProf
said...

Sounds quite like the NSF proposal I just read. The methods were solid and the data produced would be interesting, but the proposal was a huge slog (typos, sentence fragments, weird structure, odd claims of importance), and I wasn't convinced that the PIs knew what to do with the data (how to interpret it). I only gave it a Good.

It is not the reviewer's job to convince the reviewer that the proposed work is worth expending scarce resources on. That is the job of the proposer.

Too often reviewers and panels forget that.

Eli has provided reviews which go something like, IF professor X had realized Y and Z and explained A B and C this would have been a much better proposal, however, this being not the case, the proposal is somewhere between good and very good.

Somewhat the equivalent of the review reading if I had written this damn thing it would have been higher rated.

I didn't have to connect the dots for the proposers; that would be a different scenario than the one I described.

In this case, the proposal writers did a terrible job of explaining/highlighting what was important. They wrote a lot of empty phrases like "We are going to do important things" on the first page or two, and then buried the real significance later in the proposal.

FSP - I hope you at least explain in your review (very clearly without committing the same mistakes as the applicant) that it was difficult for you to see the significance of the work and that in future proposals, the applicant needs to make this more clear from the get-go. I think I was guilty of this in my first grant proposals, but the reviewers seemed to nitpick at other things which did NOT make sense to me. In response, I felt that subsequent proposals were written defensively, were likely even more confusing, and never resolved the original problem. It wasn't until I got outside help that I was able to write a clear proposal that did get funded.

I think I would have suffered a lot less (as well as subsequent reviewers) if original reviewers had provided sincere advice and suggestions.

In my opinion, a proposal should be judged based on its merit, and not on its communication. Some people are definitely worse than others when it comes to communication, and I feel they should not be penalized for it. After all, their primary job is doing the work.

That being said, it can be incredibly painful to read a poorly written proposal, and the reviewer doesn't deserved to be treated so badly. I feel this is a fixable problem. One way is for departments to devote resources to review and improve PIs proposals. This will also help PIs (particularly early career) get better with their proposal writing skills.

Another possibility is for funding agencies to warn PIs who submit good but unreadable proposals to improve their writing skills. While a poorly written proposal (with sound methods) can be tolerated once, this should not be a recurring phenomenon. If a PI who has been warned does submit another poorly written proposal, penalize the rating on that proposal because of poor writing.

I think how forgiving reviewers and panels are of shoddily written proposals depends on the panel and the general community, but also what is being proposed. You know what they say "If you have something very important to say, it doesn't matter how you say it; if you have nothing important to say, you'd better say it well." In other words, if the proposed work is really head-and-shoulders above the rest in terms of idea, creativity, how transformative it is ("transformative" is the NSF word du jour), then yeah it probably doesn't matter if it's not well written. But I have never come across one that is so much better than the rest that nothing else matters. So, for mortals who propose good, doable work, it is very important that it be well written.

"can I assume that the dense, complicated, non-linear, dry presentation of the proposed work is an indication of how the research itself will be undertaken, understood, and communicated" --I think not, but this seems like a good empirical question. Take some grants that were near the borderline score but funded ~5 years ago, ask some people to rate the clarity of the proposals, and ask some other people to rate the science that came out.

@Anon 09:21 -- Everyone can and should get "outside help" with proposals, especially early career folks. There is absolutely no substitute for having 2 - 3 colleagues critique your drafts. And the panels would be grateful, too!

A major factor in my boredom-exhaustion level when reviewing proposals has to do with the level of detail given in the proposal. Some people fill their proposals with a lot of unnecessary detail, even starting with this before explaining why this information is presented. Some detail is necessary to show that you know what you are doing, but above a certain level (10+ pages), it just seems like filling space for the sake of filling it. Does this particularly afflict those who are method-driven? I just reviewed a proposal that read like "I have this big machine so I am going to use it, and now I am going to spend 12 pages telling you how the machine works and everything I have ever done with the machine before briefly mentioning some new stuff I'd like to do."

As someone who has written both terrible and good proposals (I think they were bad because the reviewers said they were poorly written and they were good because they ended up getting funded), and as someone who teaches scientific writing regularly, I wholly disagree with Anonymous (3/19/2012 10:27:00 AM). The whole point of a proposal is to convince the reviewers and panel that your science is worth funding. Yes, bad writing will sink good science. I torpedoed a proposal 2 months ago because I just. couldn't. bring. myself. to. care. about it. The science was actually somewhat interesting, but it was so poorly described that I just didn't care. In my review, I said as much. I think you owe it to the writer to give them the reasons why the proposal was rejected. Papers will be rejected for bad writing (some of mine, even, gasp!) even if the science is good.

On the other hand, when a proposal comes across my screen that is well written (I am thinking of two people in my field in particular whose proposals and papers I've reviewed before, and they are just so well written) then it is SO MUCH easier to write a positive review. Will good writing save bad science? No, I don't think it should. Should good writing move a proposal above another of "equal scientific merit" (whatever that means)? Yes, it should, and does.

No one has mentioned this explicitly, but I believe that being a good scientist includes being able to communicate research findings effectively in a variety of settings (e.g., scientific forums, colleagues, and the general public). If one cannot communicate research results, then the research is essentially worthless. If a research proposal is poorly written, then I think it's fair to question whether results will be effectively communicated. That said, I do think most of us (particularly early-stage investigators) could use more training/guidance on grant and other scientific writing. Unfortunately, such training is usually secondary to our scientific training.

When I review proposals I am conscious of trying to review the science and not the writing. But in almost every case, a poorly written proposal makes it impossible to tell if the proposer understands what they are doing. If they don't clearly explain the proposed work and potential stumbling blocks, I cannot tell if they wrote poorly or they really do not understand what they are proposing.

The best proposal writing advice I ever got was from a senior person in my field at a Gordon conference. He told me to assume that the person reading my proposal is on a plane to Washington DC with a stack of 20 other proposals he has to read while flying coach to get to the panel. Why? Well, because the person reading my proposal was on a plane to Washington DC with a stack of 20 other proposals he has to read while flying coach to get to the panel!

About Me

I am a full professor in a physical sciences field at a large research university. I am married and have a teenaged daughter.
I have the greatest job in the world, but this will not stop me from noting some of the more puzzling and stressful aspects of my career as a science professor.
E-mail (can't promise to reply): femalescienceprofessor@gmail.com