<story><title>How not to sort by average rating (2009)</title><url>https://www.evanmiller.org/how-not-to-sort-by-average-rating.html</url></story><parent_chain><item><author>hwbehrens</author><text>While I agree with the author in principle, I think there is an implicit criteria they ignore, which is the intuitive correctness from the perspective of the user.&lt;p&gt;Imagine a user chooses &amp;quot;Sort by rating&amp;quot;, and they subsequently observe an item with an average 4.5 ranking above a score of 5.0 because it has a higher Wilson score. Some portion of users will think &amp;quot;Ah, yes, this makes sense because the 4.5 rating is based on many more reviews, therefore its Wilson score is higher.&amp;quot; and the vast, vast majority of users will think &amp;quot;What the heck? This site is rigging the system! How come this one is ranked higher than that one?&amp;quot; and erode confidence in the rankings.&lt;p&gt;In fact, these kinds of black-box rankings* frequently land sites like Yelp into trouble, because it is natural to assume that the company has a finger on the scale so to speak when it is in their financial interests to do so. In particular, entries with a higher Wilson score are likely to be more expensive because their ostensibly-superior quality commands (or depends upon) their higher cost, exacerbating this effect due to perceived-higher margins.&lt;p&gt;So the next logical step is to present the Wilson score directly, but this merely shifts the confusion elsewhere -- the user may find an item they&amp;#x27;re interested in buying, find it has one 5-star review, and yet its Wilson score is &amp;lt;&amp;lt; 5, producing at least the same perception and possibly a worse one.&lt;p&gt;Instead, providing the statistically-sound score but de-emphasizing or hiding it, such as by making it accessible in the DOM but not visible, allows for the creation of alternative sorting mechanisms via e.g. browser extensions for the statistically-minded, without sacrificing the intuition of the top-line score.&lt;p&gt;* I assume that most companies would choose not to explain the statistical foundations of their ranking algorithm.</text></item></parent_chain><comment><author>dfabulich</author><text>In another article, the author (Evan Miller) recommends not showing the average unless there are enough ratings. You would say &amp;quot;2 ratings&amp;quot; but not show the average, and just sort it wherever it falls algorithmically.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.evanmiller.org&amp;#x2F;ranking-items-with-star-ratings.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.evanmiller.org&amp;#x2F;ranking-items-with-star-ratings.h...&lt;/a&gt;&lt;p&gt;In that article, he even includes a formula for how many ratings you&amp;#x27;d need:&lt;p&gt;&amp;gt; &lt;i&gt;If you display average ratings to the nearest half-star, you probably don’t want to display an average rating unless the credible interval is a half-star wide or less&lt;/i&gt;&lt;p&gt;In my experience, the second article is more generally useful, because it&amp;#x27;s more common to sort by star rating than by thumb-up&amp;#x2F;thumb-down ranking, which is what the currently linked article is about.&lt;p&gt;And the philosophical &amp;quot;weight on the scale&amp;quot; problem isn&amp;#x27;t as bad as you&amp;#x27;d think when using these approaches. If you see an item with a perfect 5-star average and 10 reviews ranked below an item with a 4.8-star average and 1,000 reviews, and you call the sort ranking &amp;quot;sort by popularity,&amp;quot; it&amp;#x27;s pretty clear that the item with 1,000 reviews is &amp;quot;more popular.&amp;quot;</text></comment>
<story><title>How not to sort by average rating (2009)</title><url>https://www.evanmiller.org/how-not-to-sort-by-average-rating.html</url></story><parent_chain><item><author>hwbehrens</author><text>While I agree with the author in principle, I think there is an implicit criteria they ignore, which is the intuitive correctness from the perspective of the user.&lt;p&gt;Imagine a user chooses &amp;quot;Sort by rating&amp;quot;, and they subsequently observe an item with an average 4.5 ranking above a score of 5.0 because it has a higher Wilson score. Some portion of users will think &amp;quot;Ah, yes, this makes sense because the 4.5 rating is based on many more reviews, therefore its Wilson score is higher.&amp;quot; and the vast, vast majority of users will think &amp;quot;What the heck? This site is rigging the system! How come this one is ranked higher than that one?&amp;quot; and erode confidence in the rankings.&lt;p&gt;In fact, these kinds of black-box rankings* frequently land sites like Yelp into trouble, because it is natural to assume that the company has a finger on the scale so to speak when it is in their financial interests to do so. In particular, entries with a higher Wilson score are likely to be more expensive because their ostensibly-superior quality commands (or depends upon) their higher cost, exacerbating this effect due to perceived-higher margins.&lt;p&gt;So the next logical step is to present the Wilson score directly, but this merely shifts the confusion elsewhere -- the user may find an item they&amp;#x27;re interested in buying, find it has one 5-star review, and yet its Wilson score is &amp;lt;&amp;lt; 5, producing at least the same perception and possibly a worse one.&lt;p&gt;Instead, providing the statistically-sound score but de-emphasizing or hiding it, such as by making it accessible in the DOM but not visible, allows for the creation of alternative sorting mechanisms via e.g. browser extensions for the statistically-minded, without sacrificing the intuition of the top-line score.&lt;p&gt;* I assume that most companies would choose not to explain the statistical foundations of their ranking algorithm.</text></item></parent_chain><comment><author>tablespoon</author><text>&amp;gt; Imagine a user chooses &amp;quot;Sort by rating&amp;quot;, and they subsequently observe an item with an average 4.5 ranking above a score of 5.0 because it has a higher Wilson score. Some portion of users will think &amp;quot;Ah, yes, this makes sense because the 4.5 rating is based on many more reviews, therefore its Wilson score is higher.&amp;quot; and the vast, vast majority of users will think &amp;quot;What the heck? This site is rigging the system! How come this one is ranked higher than that one?&amp;quot; and erode confidence in the rankings.&lt;p&gt;It also erodes confidence in ratings when something with one fake 5 star review sorts above something else with 1000 reviews averaging 4.9.&lt;p&gt;I think you&amp;#x27;re mainly focusing on the very start of a learning curve, but eventually people get the hang of the new system.  Especially if it&amp;#x27;s named correctly (e.g. &amp;quot;sort by review-count weighted score&amp;quot;).</text></comment>