I'm going to be developing some functionality that will crawl various public web sites and process/aggregate the data on them.  Nothing sinister like looking for e-mail addresses - in fact it's something that might actually drive additional traffic to their sites.  But I digress.Other than honouring robots.txt, are there any rules or guidelines, written or unwritten, that I ought to be following in order to (a) avoid appearing malicious and potentially being banned, and (b) not cause any problems for the site owners/webmasters?Some examples I can think of which may or may not matter:Number of parallel requestsTime between requestsTime between entire crawlsAvoiding potentially destructive links (don't want to be the Spider of Doom - but who knows if this is even practical)That's really just spit-balling, though; is there any tried-and-tested wisdom out there that's broadly applicable for anybody who intends to write or utilize a spider?
How to be a good citizen when crawling web sites?
web scraping;web crawler
Besides obeying robots.txt, obey nofollow and noindex in <meta> elements and links:There are many who believe robots.txt is not the proper way to block indexing and because of that viewpoint, have instructed many site owners to rely on the <meta name=robots content=noindex> tag to tell web crawlers not to index a page.If you're trying to make a graph of connections between websites (anything similar to PageRank),  (and <meta name=robots content=nofollow>) is supposed to indicate the source site doesn't trust the destination site enough to give it a proper endorsement. So while you can index the destination site, you ought not store the relation between the two sites.SEO is more of an art than a real science, and it's practiced by a lot of people who know what they're doing, and a lot of people who read the executive summaries of people who know what they're doing. You're going to run into issues where you'll get blocked from sites for doing things that other sites found perfectly acceptable due to some rule someone overheard or read in a blog post on SEOmoz that may or may not be interpreted correctly.Because of that human element, unless you are Google, Microsoft, or Yahoo!, you are presumed malicious unless proven otherwise. You need to take extra care to act as though you are no threat to a web site owner, and act in accordance with how you would want a potentially malicious (but hopefully benign) crawler to act:stop crawling a site once you detect you're being blocked: 403/401s on pages you know work, throttling, time-outs, etc. avoid exhaustive crawls in relatively short periods of time: crawl a portion of the site, and come back later on (a few days later) to crawl another portion. Don't make parallel requests.avoid crawling potentially sensitive areas: URLs with /admin/ in them, for example.Even then, it's going to be an up-hill battle unless you resort to black-hat techniques like UA spoofing or purposely masking your crawling patterns: many site owners, for the same reasons above, will block an unknown crawler on sight instead of taking the chance that there's someone not trying to hack their site. Prepare for a lot of failure. One thing you could do to combat the negative image an unknown crawler is going to have is to make it clear in your user-agent string who you are:Aarobot Crawler 0.9 created by John Doe. See http://example.com/aarobot.html for more information.Where http://example.com/aarobot.html explains what you're trying to accomplish and why you're not a threat. That page should have a few things:Information on how to contact you directlyInformation about what the crawler collects and why it's collecting itInformation on how to opt-out and have any data collected deletedThat last one is key: a good opt-out is like a Money Back Guarantee and scores an unreasonable amount of goodwill. It should be humane: one simple step (either an email address or, ideally, a form) and comprehensive (there shouldn't be any gotchas: opt-out means you stop crawling without exception).