Why does NIH study section website robot.txt disallow search? NIH posts their study section members who review grants online (e.g. Aging Systems and Geriatrics Study Section (ASG)'s roster is provided here). I was trying to search the study sections to find whether a particular reviewer served as a study section member. However, this is not easy, because their website does not provide a search function that does this (as far as I can tell), and the website NIH study section rosters that contain grant reviewer names disallow search engines from crawling and indexing them via robots.txt.

http://internet.csr.nih.gov/robots.txt

User-agent: * 
Disallow: /


Clearly, I can do a manual search by going to each page and finding if any of the study section contains the name I'm looking for, or manually make a database for personal use.

What are the possible rationales for disallowing search engine indexing reviewers?
@Memming, this could be done for any number of reasons.  The two I can think of off the top of my head is 1) to prevent site slowdowns especially due to what is likely to be the duplication of information developed by indexing the NIH site and 2) to protect (to some extent) the privacy of the participants.

One suggestion, have you tried using Google to search the site?  Search is not the same as indexing.  So if you're looking for sections with "Joe Brown" you can try searching "Joe Brown site:csr.nih.gov".  I was able to get results with this approach using a name I picked at random from the study sessions.