I've seen quite a lot of tutorials that recommend you to ban user agents containing the strings libwww-perl and msnbot. I understand why one would ban libwww-perl, it's mainly if not only used for hacking and spamming.But why are there so many sites recommending to ban msnbot/bingbot?Since it's a search engine, even if only with a marginal market share, I would except one would want this bot to crawl one's sites.What is it that msnbot does that makes people ban it?
Why deny access to website for msnbot/bingbot?
seo;web crawlers;bing
I don't think people should ban bing bot.Bing has an equivalent Bing Webmaster tools at http://www.bing.com/toolbox/webmaster/where they also have 'Crawl Settings' where you can adjust the crawl rate as seen in this video:http://www.bing.com/videos/watch/video/bing-webmaster-tools-crawl-rate-settings/1ii1ej9jzGooglebot is just as notorious in excessive crawling of sites as msnbot. Also the better (traffic/linkage) your site gets, the more googlebot crawls. Just look how fast stackexchange questions get indexed after being posted. You can see how much these bots hit up your server if you check your access logs.I also discovered that msnbot supports robots.txt Crawl-delay parameter.http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx