MP3 Crawler
deutronium: You may be interested in Nutch (http://lucene.apache.org/nutch/), an Open Source crawler, which handles indexing etc. for you.  Its also based on Hadoop, so it should scale nicely just by throwing more machines at the job.