<story><title>20% of requests for Wikimedia Commons are for one image of a flower</title><url>https://phabricator.wikimedia.org/T273741</url></story><parent_chain><item><author>zelon88</author><text>&amp;gt; Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before)&lt;p&gt;Do you have a source for this?&lt;p&gt;I&amp;#x27;d be willing to bet that the ONLY reason why 15% of their daily queries &amp;quot;haven&amp;#x27;t been seen before&amp;quot; is because they add un-needed complexity like fingerprinting. You&amp;#x27;re making it seem like they&amp;#x27;ve never seen a query for &amp;quot;cute animals&amp;quot; before when obviously they have. They choose to do a lot of extra leg work because of who you are.&lt;p&gt;So your claim that 15% of their queries have &amp;quot;never been seen before&amp;quot; is probably inaccurate. I&amp;#x27;d be willing to bet that &amp;quot;15% of their queries are unique because of the user, location, or other external factor separate from the query itself.&amp;quot;&lt;p&gt;They&amp;#x27;ve seen your query before. They&amp;#x27;ve just never seen &lt;i&gt;you&lt;/i&gt; make this query from this device on this side of town before.</text></item><item><author>nostrademons</author><text>Search is also largely read-only.  The advantage Wikipedia has is that its traffic overwhelming goes to the head of the page distribution, so simple caching solutions work very well.  Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before), and so needs to do a lot of computation per query.</text></item><item><author>ShakataGaNai</author><text>Wikimedia&amp;#x27;s infrastructure is radically different than most FAANG.&lt;p&gt;In large part because 99% (+&amp;#x2F;-) of their traffic is read only. While Facebook and Google have to do heavy workloads for every click and action taken on their services, Wikimedia can cache basically everything. Allowing them to operate on a tiny fraction of the number of machines (and infrastructure) that the rest of the players do.</text></item><item><author>IfOnlyYouKnew</author><text>Wikimedia is unique in running some of the most popular websites with open access to almost all systems. As someone who has never been on the inside of FAANG, I found it rather interesting to browse around the backend infrastructure.&lt;p&gt;See, for example, their statistics at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;grafana.wikimedia.org&amp;#x2F;d&amp;#x2F;000000102&amp;#x2F;production-logging?orgId=1&amp;amp;refresh=5m&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;grafana.wikimedia.org&amp;#x2F;d&amp;#x2F;000000102&amp;#x2F;production-logging...&lt;/a&gt;</text></item></parent_chain><comment><author>tylerhou</author><text>If you took into account user, location, etc. 15% seems too low. I almost never search for the exact same thing twice in the same location.&lt;p&gt;15% of the queries themselves are unique. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;blog.google&amp;#x2F;products&amp;#x2F;search&amp;#x2F;our-latest-quality-improvements-search&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;blog.google&amp;#x2F;products&amp;#x2F;search&amp;#x2F;our-latest-quality-impro...&lt;/a&gt;&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.google.com&amp;#x2F;search&amp;#x2F;howsearchworks&amp;#x2F;responses&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.google.com&amp;#x2F;search&amp;#x2F;howsearchworks&amp;#x2F;responses&amp;#x2F;&lt;/a&gt;&lt;p&gt;I work for Google (and used to work on Search).</text></comment>
<story><title>20% of requests for Wikimedia Commons are for one image of a flower</title><url>https://phabricator.wikimedia.org/T273741</url></story><parent_chain><item><author>zelon88</author><text>&amp;gt; Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before)&lt;p&gt;Do you have a source for this?&lt;p&gt;I&amp;#x27;d be willing to bet that the ONLY reason why 15% of their daily queries &amp;quot;haven&amp;#x27;t been seen before&amp;quot; is because they add un-needed complexity like fingerprinting. You&amp;#x27;re making it seem like they&amp;#x27;ve never seen a query for &amp;quot;cute animals&amp;quot; before when obviously they have. They choose to do a lot of extra leg work because of who you are.&lt;p&gt;So your claim that 15% of their queries have &amp;quot;never been seen before&amp;quot; is probably inaccurate. I&amp;#x27;d be willing to bet that &amp;quot;15% of their queries are unique because of the user, location, or other external factor separate from the query itself.&amp;quot;&lt;p&gt;They&amp;#x27;ve seen your query before. They&amp;#x27;ve just never seen &lt;i&gt;you&lt;/i&gt; make this query from this device on this side of town before.</text></item><item><author>nostrademons</author><text>Search is also largely read-only.  The advantage Wikipedia has is that its traffic overwhelming goes to the head of the page distribution, so simple caching solutions work very well.  Google has a pretty extreme long-tail distribution (~15% of daily queries have never been seen before), and so needs to do a lot of computation per query.</text></item><item><author>ShakataGaNai</author><text>Wikimedia&amp;#x27;s infrastructure is radically different than most FAANG.&lt;p&gt;In large part because 99% (+&amp;#x2F;-) of their traffic is read only. While Facebook and Google have to do heavy workloads for every click and action taken on their services, Wikimedia can cache basically everything. Allowing them to operate on a tiny fraction of the number of machines (and infrastructure) that the rest of the players do.</text></item><item><author>IfOnlyYouKnew</author><text>Wikimedia is unique in running some of the most popular websites with open access to almost all systems. As someone who has never been on the inside of FAANG, I found it rather interesting to browse around the backend infrastructure.&lt;p&gt;See, for example, their statistics at &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;grafana.wikimedia.org&amp;#x2F;d&amp;#x2F;000000102&amp;#x2F;production-logging?orgId=1&amp;amp;refresh=5m&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;grafana.wikimedia.org&amp;#x2F;d&amp;#x2F;000000102&amp;#x2F;production-logging...&lt;/a&gt;</text></item></parent_chain><comment><author>bagels</author><text>&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;blog.google&amp;#x2F;products&amp;#x2F;search&amp;#x2F;our-latest-quality-improvements-search&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;blog.google&amp;#x2F;products&amp;#x2F;search&amp;#x2F;our-latest-quality-impro...&lt;/a&gt;&lt;p&gt;&amp;quot;There are trillions of searches on Google every year. In fact, 15 percent of searches we see every day are new&amp;quot;</text></comment>