<story><title>Ask HN: Why doesn&apos;t anyone create a search engine comparable to 2005 Google?</title><text>I seem to recall that Google consistently produced relevant results and strictly respected search operators in 2005 (?), unlike the modern Google. And back then, I think search results were the same for everyone, rather than being customized for each user.</text></story><parent_chain><item><author>JacobThreeThree</author><text>I think people also have an inflated recollection of how good Google actually was back in 2005.&lt;p&gt;Back then Google was only going up against indexes and link-rings, not 2021 Google&amp;#x2F;Bing&amp;#x2F;DDG&amp;#x2F;etc.</text></item><item><author>SamBam</author><text>&amp;gt; You do a search on Gigablast and say, well, why didn&amp;#x27;t it get this result that Google got. And that&amp;#x27;s because the index isn&amp;#x27;t big enough&lt;p&gt;I wionder how much this is true, and how much (despite all our rhetoric to the contrary) it&amp;#x27;s because we have actually come to expect Google&amp;#x27;s modern proprietary page ranking, which counts more than just inbound links but all sorts of other signals (freshness, relevance to our previous queries, etc.).&lt;p&gt;We dislike the additional signals when it feels like Google is trying to second-guess our intentions, but we probably don&amp;#x27;t notice how well they work when they give us the result we expect in the first three links.</text></item><item><author>gbmatt</author><text>Ha, yes, I&amp;#x27;ve done that at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;gigablast.com&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;gigablast.com&amp;#x2F;&lt;/a&gt; .
The biggest problems now are the following:
1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages.
2) Hardware costs are too high. It&amp;#x27;s much more expensive now to build a large index (50B+ pages) to be competitive.&lt;p&gt;I believe my algorithms are decent, but the biggest problem for Gigablast is now the index size. You do a search on Gigablast and say, well, why didn&amp;#x27;t it get this result that Google got. And that&amp;#x27;s because the index isn&amp;#x27;t big enough because I don&amp;#x27;t have the cash for the hardware. btw, I&amp;#x27;ve been working on this engine for over 20 years and have coded probably 1-2M lines of code on it.</text></item></parent_chain><comment><author>eitland</author><text>&amp;gt; I think people also have an inflated recollection of how good Google actually was back in 2005.&lt;p&gt;I&amp;#x27;ve been pointing this out for at least close to a decade.&lt;p&gt;I know since I bothered to screenshot and blog about it in 2012.&lt;p&gt;I&amp;#x27;ll admit mistakes happened back then too, but they were more forgivable like keyword stuffing on unrelated pages. Back then Google were on our side and removed those as fast as possible.&lt;p&gt;Today however the problem isn&amp;#x27;t that someone hss stuffed the keyword into an unrelated page but that Google themselves mix a whole lot of completely irrelevant pages into the results, probably because some metrics go up when they do that.&lt;p&gt;Thinking about it it seems logical that for a search engine that practically speaking has monopoly both on users and as mattgb points out - tonsome degree also on indexing - serving the correct answer first is just dumb: if they can keep me going between their search results and  tech blogs with their ads embedded one, two or five times extra that means one, two or five times more ad impressions.&lt;p&gt;Note that I&amp;#x27;m not necessarily suggesting an grand evil master plan here, only that end-to-end metrics will improve as long as there is no realistic competition.</text></comment>
<story><title>Ask HN: Why doesn&apos;t anyone create a search engine comparable to 2005 Google?</title><text>I seem to recall that Google consistently produced relevant results and strictly respected search operators in 2005 (?), unlike the modern Google. And back then, I think search results were the same for everyone, rather than being customized for each user.</text></story><parent_chain><item><author>JacobThreeThree</author><text>I think people also have an inflated recollection of how good Google actually was back in 2005.&lt;p&gt;Back then Google was only going up against indexes and link-rings, not 2021 Google&amp;#x2F;Bing&amp;#x2F;DDG&amp;#x2F;etc.</text></item><item><author>SamBam</author><text>&amp;gt; You do a search on Gigablast and say, well, why didn&amp;#x27;t it get this result that Google got. And that&amp;#x27;s because the index isn&amp;#x27;t big enough&lt;p&gt;I wionder how much this is true, and how much (despite all our rhetoric to the contrary) it&amp;#x27;s because we have actually come to expect Google&amp;#x27;s modern proprietary page ranking, which counts more than just inbound links but all sorts of other signals (freshness, relevance to our previous queries, etc.).&lt;p&gt;We dislike the additional signals when it feels like Google is trying to second-guess our intentions, but we probably don&amp;#x27;t notice how well they work when they give us the result we expect in the first three links.</text></item><item><author>gbmatt</author><text>Ha, yes, I&amp;#x27;ve done that at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;gigablast.com&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;gigablast.com&amp;#x2F;&lt;/a&gt; .
The biggest problems now are the following:
1) Too hard to spider the web. Gatekeeper companies like Cloudflare (owned in part by Google) and Cloudfront make it really difficult for upstart search engines to download web pages.
2) Hardware costs are too high. It&amp;#x27;s much more expensive now to build a large index (50B+ pages) to be competitive.&lt;p&gt;I believe my algorithms are decent, but the biggest problem for Gigablast is now the index size. You do a search on Gigablast and say, well, why didn&amp;#x27;t it get this result that Google got. And that&amp;#x27;s because the index isn&amp;#x27;t big enough because I don&amp;#x27;t have the cash for the hardware. btw, I&amp;#x27;ve been working on this engine for over 20 years and have coded probably 1-2M lines of code on it.</text></item></parent_chain><comment><author>pbhjpbhj</author><text>2005? There were loads of other search engines (SE), and many meta-SE: hotbot, dogpile, metacrawler, ... (IIRC), plenty more.&lt;p&gt;There was also indexes, which Yahoo, AOL (remember them!) had but there was, what was it called, dmoz?, the open web directory. When Google started, being in the right web directory gave you a boost in SERPs as it was used as a domain trust indicator, and the categories were used for keywords. Of course it got gamed hard.&lt;p&gt;Google was good, but I used it as an alt for maybe 6 months before it won over my main SE at the time. I&amp;#x27;ve tried but can&amp;#x27;t remember what SE that was, Omni-something??&lt;p&gt;One of the main things Google had was all the extra operators like link: inurl:, etc., but they had Boolean logic operators too at one point I think.</text></comment>