<story><title>How Not To Sort By Average Rating</title><url>http://evanmiller.org/how-not-to-sort-by-average-rating.html</url></story><parent_chain><item><author>_delirium</author><text>Here&apos;s a paper proposing a solution in that space, and which also compares itself to the article linked here (kind of nice to see... papers sometimes fail to cite stuff that&apos;s &quot;only&quot; posted online rather than properly published, even if the authors know about it and it&apos;s quite relevant): &lt;a href=&quot;http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2011.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...&lt;/a&gt;&lt;p&gt;I emailed Miller a while ago to see what he thought of this reply, and he thought it also seemed like a reasonable approach. But, in his view, the criticisms of his method within their framework include things that in practice he sees as features. In particular, they view the bias caused by using the &lt;i&gt;lower&lt;/i&gt; bound as a bug, but he prefers rankings to be be &quot;risk-averse&quot; in recommending, avoiding false positives more than false negatives. Of course, that biased preference could also be encoded explicitly in a more complex Bayesian setup, which would also be a bit more principled, since you could directly choose the degree of bias, instead of indirectly choosing it via your choice of confidence level on the Wilson score interval.</text></item><item><author>NathanRice</author><text>While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer.&lt;p&gt;Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior.  This gives you a posterior distribution over new ratings for an item.  The benefit of a posterior here is that it lets you rank items by thinking in terms of the probability that the viewer would rank one item higher than another at random.  By adjusting the magnitude of the alpha parameter to the dirichlet prior, you adjust your sensitivity to small numbers of observations.  A small initial alpha will lead to rapid changes in the posterior upon observing ratings, whereas a large alpha requires a significant body of evidence.&lt;p&gt;The best part of the multinomial model with conjugate dirichlet prior is that the math is REALLY simple.  The normalizing constant for the dirichlet distribution looks scary when stated in terms of the gamma function, but given this is the discrete case, just pretend everywhere you see the gamma(x), it is replaced with (x - 1)! and you will be ok.&lt;p&gt;Let me know if you would like to learn more, I would be happy to help.</text></item></parent_chain><comment><author>NathanRice</author><text>I don&apos;t think you have to resort to any overly complex machinery to achieve similar behavior.  The simplest approach is to just use a non uniform prior.  His pessimistic bound could be emulated by having an initial alpha that places more weight on low star ratings.  The intuitive interpretation of that being &quot;things are probably bad unless proven good&quot; roughly.  Another option would be to generate the prior based on the posterior distributions of other items.  Just take the distribution of ratings observations for all products of a given type (perhaps only items produced by that company?) to get a sensible prior on a new item in that category.&lt;p&gt;The strength of priors here is that it is very easy to take intuitions and encode them statistically, in an understandable way.  Taking the lower bound of a test statistic doesn&apos;t admit much in the way of intuition.</text></comment>
<story><title>How Not To Sort By Average Rating</title><url>http://evanmiller.org/how-not-to-sort-by-average-rating.html</url></story><parent_chain><item><author>_delirium</author><text>Here&apos;s a paper proposing a solution in that space, and which also compares itself to the article linked here (kind of nice to see... papers sometimes fail to cite stuff that&apos;s &quot;only&quot; posted online rather than properly published, even if the authors know about it and it&apos;s quite relevant): &lt;a href=&quot;http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2011.pdf&quot; rel=&quot;nofollow&quot;&gt;http://www.dcs.bbk.ac.uk/~dell/publications/dellzhang_ictir2...&lt;/a&gt;&lt;p&gt;I emailed Miller a while ago to see what he thought of this reply, and he thought it also seemed like a reasonable approach. But, in his view, the criticisms of his method within their framework include things that in practice he sees as features. In particular, they view the bias caused by using the &lt;i&gt;lower&lt;/i&gt; bound as a bug, but he prefers rankings to be be &quot;risk-averse&quot; in recommending, avoiding false positives more than false negatives. Of course, that biased preference could also be encoded explicitly in a more complex Bayesian setup, which would also be a bit more principled, since you could directly choose the degree of bias, instead of indirectly choosing it via your choice of confidence level on the Wilson score interval.</text></item><item><author>NathanRice</author><text>While I agree with the spirit of the article, this is one of those cases where a Bayesian treatment is conceptually much clearer.&lt;p&gt;Assume that ratings are being generated by a stable stochastic process where the underlying distribution is multinomial (ignoring the ordinal character of ratings, for the time being) and use a dirichlet conjugate prior.  This gives you a posterior distribution over new ratings for an item.  The benefit of a posterior here is that it lets you rank items by thinking in terms of the probability that the viewer would rank one item higher than another at random.  By adjusting the magnitude of the alpha parameter to the dirichlet prior, you adjust your sensitivity to small numbers of observations.  A small initial alpha will lead to rapid changes in the posterior upon observing ratings, whereas a large alpha requires a significant body of evidence.&lt;p&gt;The best part of the multinomial model with conjugate dirichlet prior is that the math is REALLY simple.  The normalizing constant for the dirichlet distribution looks scary when stated in terms of the gamma function, but given this is the discrete case, just pretend everywhere you see the gamma(x), it is replaced with (x - 1)! and you will be ok.&lt;p&gt;Let me know if you would like to learn more, I would be happy to help.</text></item></parent_chain><comment><author>jules</author><text>Can anybody replicate their results at Proposition 5? My tests contradict their conclusion, namely that the total score is not monotonic (i.e. when I test I do get that the total score is monotonic).</text></comment>