<story><title>Ask HN: I want to dive into how to make search engines</title><text>But I don&amp;#x27;t know where to start or what to study. If I were going for masters in this subject, what would my courses be?</text></story><parent_chain><item><author>jasode</author><text>&lt;i&gt;&amp;gt;search engines&lt;/i&gt;&lt;p&gt;You can decompose a &amp;quot;search engine&amp;quot; into multiple big components and figure out what you want to look at first:&lt;p&gt;(1) web crawler&amp;#x2F;spiders&lt;p&gt;(2) database cache of web content -- aka building the &amp;quot;search index&amp;quot;&lt;p&gt;(3) algorithm of scoring&amp;#x2F;weighing&amp;#x2F;ranking of pages -- e.g. &amp;quot;PageRank&amp;quot;&lt;p&gt;(4) query engine -- translating user inputs into returning the most &amp;quot;relevant&amp;quot; pages&lt;p&gt;Each technical topic is a sub-specialty and can be staffed by dedicated engineers.  There are also more topics such as lexical analysis, distributed computing (for all 4 areas), etc.&lt;p&gt;If you&amp;#x27;re mainly focused on experimenting with programming another ranking algorithm, you can skip part (1) by leveraging the dataset from Common Crawl: 
 &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;index.commoncrawl.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;index.commoncrawl.org&amp;#x2F;&lt;/a&gt;&lt;p&gt;Here are some videos about PageRank:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=JGQe4kiPnrU&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=JGQe4kiPnrU&lt;/a&gt; , &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=qxEkY8OScYY&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=qxEkY8OScYY&lt;/a&gt;&lt;p&gt;... but keep in mind that the scope of those videos omits all of (1), (2), and (4).</text></item></parent_chain><comment><author>swyx</author><text>I would also require reading on the major OSS search engines, e.g. Lucene, Solr and then Elasticsearch.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Apache_Lucene&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Apache_Lucene&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Apache_Solr&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Apache_Solr&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Elasticsearch&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;en.wikipedia.org&amp;#x2F;wiki&amp;#x2F;Elasticsearch&lt;/a&gt;&lt;p&gt;those who dont know history will repeat it, etc&lt;p&gt;then maybe the newer stuff like &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;typesense.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;typesense.org&amp;#x2F;&lt;/a&gt;&lt;p&gt;to be clear i dont know these things but thats what I would do and I&amp;#x27;d happily read fly.io style blogposts drip feeding out knowledge over time</text></comment>
<story><title>Ask HN: I want to dive into how to make search engines</title><text>But I don&amp;#x27;t know where to start or what to study. If I were going for masters in this subject, what would my courses be?</text></story><parent_chain><item><author>jasode</author><text>&lt;i&gt;&amp;gt;search engines&lt;/i&gt;&lt;p&gt;You can decompose a &amp;quot;search engine&amp;quot; into multiple big components and figure out what you want to look at first:&lt;p&gt;(1) web crawler&amp;#x2F;spiders&lt;p&gt;(2) database cache of web content -- aka building the &amp;quot;search index&amp;quot;&lt;p&gt;(3) algorithm of scoring&amp;#x2F;weighing&amp;#x2F;ranking of pages -- e.g. &amp;quot;PageRank&amp;quot;&lt;p&gt;(4) query engine -- translating user inputs into returning the most &amp;quot;relevant&amp;quot; pages&lt;p&gt;Each technical topic is a sub-specialty and can be staffed by dedicated engineers.  There are also more topics such as lexical analysis, distributed computing (for all 4 areas), etc.&lt;p&gt;If you&amp;#x27;re mainly focused on experimenting with programming another ranking algorithm, you can skip part (1) by leveraging the dataset from Common Crawl: 
 &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;index.commoncrawl.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;index.commoncrawl.org&amp;#x2F;&lt;/a&gt;&lt;p&gt;Here are some videos about PageRank:&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=JGQe4kiPnrU&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=JGQe4kiPnrU&lt;/a&gt; , &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=qxEkY8OScYY&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.youtube.com&amp;#x2F;watch?v=qxEkY8OScYY&lt;/a&gt;&lt;p&gt;... but keep in mind that the scope of those videos omits all of (1), (2), and (4).</text></item></parent_chain><comment><author>abadger9</author><text>As someone who works in this space, ^ this. I would say don&amp;#x27;t overthink any component, use common crawl (&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;commoncrawl.org&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;commoncrawl.org&amp;#x2F;&lt;/a&gt;) to build your initial index, use a pagerank implementation that&amp;#x27;s been thoroughly researched and published, and use off the shelf components from the apache foundation when you can.</text></comment>