Would you buy datasets of Web authoring methods & the Web's structure?
olalonde: What would be interesting would be to let users hook their own custom functions to your web crawler. In other words, whenever you crawl a link, you feed the link to the developer's script which does whatever it wants with it. I've had plenty of ideas that would involve a web crawler, but I just don't have the time to customize an open source crawler. There are too many complications like not falling into infinite loops (such as a calendar type pages).