Anyone here do Web mining/harvesting/scraping?
truiu: I have done some webscraping with twisted and BeautifulSoup. But it was just for a few small sites.That was when I found http://scrapy.org/ , a complete python framework for writing webcrawlers. It looks promissing, but I haven't used it so far.