MP3 Crawler
jm4: They probably use a better crawler than the one you've put together. Reliable crawling is not the easiest problem to solve. There are a lot of crappy sites out there. When you're Google you can tell them to screw off. When you're small and you need to crawl the content you have to figure out a way to make things work.To accurately collect links you've got to be able to follow redirects (this is really a no brainer), interpret JavaScript, handle DOM events, have AJAX support, possibly parse Flash files for content or links, etc. There are still plenty of sites out there that use Flash for navigation and don't provide a fallback. I recently saw a site that used the window.onload event to call a function that wrote out the HTML for the entire page using document.write.Depending on what your needs are you could end up with anything from a small script to a full fledged browser. You could either develop something yourself, use an open source crawler or script Mozilla or IE. With a couple Perl modules you could have your own headless Mozilla.Once you have a good crawler it's still going to be tricky to use. There are all sorts of spider traps out there- circular navigation, unique URLs that produce duplicate content, etc. Sometimes it's deliberate; most of the time it's not. People just don't usually design sites with web crawlers in mind. It may taking a little prodding (site-specific configuration) to make it work.