Ethics and laws regarding scraping websites?
dhotson: Apart from the copyright issues, if you're going to scrape or crawl someone's site.. it's best to be polite about it.I wrote a web crawler a few years back and at the time I didn't really understand the implications of having a crawler grab 20+ pages concurrently from a site.I learned pretty fast when I found a few sites had banned my crawler.. Oops, sorry guys!