Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says:Unknown robot (identified by 'bot*') consumed 164 GB this month.By comparison, Googlebot consumed 10 GB and visitors (viewed traffic) consumed 25 GB.This means rogue bots are consumming over 6X the bandwidth of visitors. For other siteswhich I run (about a dozen) the normal ratio is 25%, so for 25GB of viewed traffic, bots take about 6GB in TOTAL.The question therefore is: How to identify which bot(s) are causing this huge amount of request and how to stop them or slow them down if they are useful?Obviously, most bots that visit the site are important including the Googlebot, Yahoo Slurp, MSNBot, etc, including the AdSense/DoubleClick bots, so I cannot simply block all bots.The reason I am investigating this is that I am reaching the limit of bandwidth and exceeded CPU usage for my host, so I was sent a notice.
How To Track Down and Stop Rogue Bots?
web crawlers;bandwidth
Create a page that captures IP addresses of anyone who visits it. Add those IPs to an htaccess file that blocks it. (see example here)Link to that page in the footer of your website using 1px transparent imageBlock that page in robots.txt so good robots won't find itNote: whitelisting good IPs and/or useragents is also a good idea (IP Addresses of Search Engine Spiders)