While attempting to block bots can help in free up resources and clean up your logs it’s important to note that robots.txt and even using the meta tag on pages noindex does not actually stop bots visiting your site. They can still crawl your site occasionally to see if the denied from robots has been removed. A lot of bots don’t even use a user agent and will use a standard user agent. The bots I’m referring to are typically SEO harvesting bots that scan for backlinks and not the general ones you find from search engines. 

Rather than blocking the bots you should just factor in these bots when counting up your visitors, after a while of actively monitoring your site your establish a rough figure which are bots. Most people care about unique visits and this rules out the bots since they are constantly returning. In this day and age there are plenty of servers, shared hosting that can handle these bots so other than pages that you don’t want indexed I see no reason why to block these types of bots. Of course you have harmful bots as well but these certainly won’t use user agent ;).

Personally I believe blocking robots is a waste of time as they don’t use that much resources at all, SEO robots can help as they list your site on PR0 pages which of course increases your PageRank and there automated so you won’t get punished by them. 

Logs Issue

You should use a proper log viewer that enables you to filter out certain requests, this makes it easier when reviewing your logs. Good viewers can filter out lots of things like normal visits, 404s' and so forth.

Should I block bots from my site and why?
I find myself always purchasing games that have similar game mechanics. I prefer worker-placement games, and railroad tycoon type games. All of these games borrow from each other in terms of how the game plays. I find this type of game suits my strategic style of play.

I now want to introduce something new to my gaming group, and want to know what type of game mechanics I am really missing out on, and why they add to the overall experience of the game.

What I want to know is what other type of game mechanics make a game more interesting to play, increase the strategic element and really make the game what it is. The type of thing I am after is similar comments to a conversation at my last gaming meeting, which was


  I much prefer Age of Steam to Railroad Tycoon, because of the way the resources are replenished. Where as Railroad Tycoon can dry up a city's resource quite quickly.

