I would like to somehow be able to block all search engines except Google, Yahoo & Bing (and their related sites like Google Images) from crawling my site as they consume a lot of server and bandwidth but don't bring any traffic. Is this easily done or difficult? It would be good if someone maintained a list of small search engines that could be pasted into a robots.txt file to block them.Also, I realize I cannot block crawlers that ignore the robots.txt or sites from surreptitiously scraping and crawling, but that is not what I want. I just want to block all the Altavistas, Hotbots, Lycos (do these even still exist) and the university experiment crawlers from wasting my time.
Blocking all search engines except the big ones
web crawlers