What strategy would you take to build a search engine today?
jacquesm: I would aggregate all the knowledge in linkdumps / fora where voting is possible, possibly seed the database by creating a giant repository of links such as a social bookmarking site.It would get built up slowly but it would be of a higher quality than you could probably reach with crawling. Effectively you'd be crowdsourcing the ratings system of your search engine.A karma system would keep the spammers out, or at least have them identified fast enough.Of course there are plenty of things that would need to be fleshed out wrt to abuse potential and 'gaming the system' but I think it would stand a fighting chance.Crawling the web is just going to get you tons of garbage, comparable with moving the contents of the local trash dump into your house because you know there must be a pair of earrings in there somewhere.There will come a time when 'pagecount' is not the defining measure of search engine quality. Google seems to be on to that because they've dropped their 'indexing xxx pages' message from the homepage long ago.I would also allow users to 'hide' certain domains from their future search results, and use that to help identifying sites that contain mostly (or exclusively) trash.