<story><title>Most Winning A/B Test Results Are Illusory [pdf]</title><url>http://www.qubit.com/sites/default/files/pdf/mostwinningabtestresultsareillusory_0.pdf</url></story><parent_chain><item><author>ted_dunning</author><text>This is yet another article that ignores the fact that there is a MUCH better approach to this problem.&lt;p&gt;Thompson sampling avoids the problems of multiple testing, power, early stopping and so on by starting with a proper Bayesian approach. The idea is that the question we want to answer is more &amp;quot;Which alternative is nearly as good as the best with pretty high probability?&amp;quot;. This is very different from the question being answered by a classical test of significance. Moreover, it would be good if we could answer the question partially by decreasing the number of times we sample options that are clearly worse than the best. What we want to solve is the multi-armed bandit problem, not the retrospective analysis of experimental results problem.&lt;p&gt;The really good news is that Thompson sampling is both much simpler than hypothesis testing can be done in far more complex situations. It is known to be an asymptotically optimal solution to the multi-armed bandit and often takes only a few lines of very simple code to implement.&lt;p&gt;See &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;tdunning.blogspot.com&amp;#x2F;2012&amp;#x2F;02&amp;#x2F;bayesian-bandits.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;tdunning.blogspot.com&amp;#x2F;2012&amp;#x2F;02&amp;#x2F;bayesian-bandits.html&lt;/a&gt; for an essay and see &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;tdunning&amp;#x2F;bandit-ranking&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;tdunning&amp;#x2F;bandit-ranking&lt;/a&gt; for an example applied to ranking.</text></item></parent_chain><comment><author>yummyfajitas</author><text>Thompson sampling is a great tool. I&amp;#x27;ve used it to make reasonably large amounts of money. But it does not solve the same problem as A&amp;#x2F;B testing.&lt;p&gt;Thompson Sampling (at least the standard approach) assumes that conversion rates do not change. In reality they vary significantly over a week, and this fundamentally breaks bandit algorithms.&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.chrisstucchio.com&amp;#x2F;blog&amp;#x2F;2015&amp;#x2F;dont_use_bandits.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.chrisstucchio.com&amp;#x2F;blog&amp;#x2F;2015&amp;#x2F;dont_use_bandits.htm...&lt;/a&gt;&lt;p&gt;Furthermore, you do not need to use Thompson Sampling to have a proper Bayesian approach. At VWO we also use a proper Bayesian approach, but we use A&amp;#x2F;B testing in to avoid the various pitfalls that Thompson Sampling has. Google Optimize uses an approach very similar to ours, (although it may be flawed [1]) and so does A&amp;#x2F;B Tasty (probably not flawed).&lt;p&gt;&lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;cdn2.hubspot.net&amp;#x2F;hubfs&amp;#x2F;310840&amp;#x2F;VWO_SmartStats_technical_whitepaper.pdf&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;cdn2.hubspot.net&amp;#x2F;hubfs&amp;#x2F;310840&amp;#x2F;VWO_SmartStats_technic...&lt;/a&gt;&lt;p&gt;Note: I&amp;#x27;m the Director of Data Science at VWO. Obviously I&amp;#x27;m biased, etc. However my post critiquing bandits was published before I took on this role. It was a followup to a previous post of mine which led people to accidentally misuse bandits: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.chrisstucchio.com&amp;#x2F;blog&amp;#x2F;2012&amp;#x2F;bandit_algorithms_vs_ab.html&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.chrisstucchio.com&amp;#x2F;blog&amp;#x2F;2012&amp;#x2F;bandit_algorithms_vs...&lt;/a&gt;&lt;p&gt;[1] The head of data science at A&amp;#x2F;B Tasty suggests Google Optimize counts &lt;i&gt;sessions&lt;/i&gt; rather than &lt;i&gt;visitors&lt;/i&gt;, which would break the IID assumption. &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.abtasty.com&amp;#x2F;uk&amp;#x2F;blog&amp;#x2F;data-scientist-hubert-google-optimize&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.abtasty.com&amp;#x2F;uk&amp;#x2F;blog&amp;#x2F;data-scientist-hubert-google...&lt;/a&gt;</text></comment>
<story><title>Most Winning A/B Test Results Are Illusory [pdf]</title><url>http://www.qubit.com/sites/default/files/pdf/mostwinningabtestresultsareillusory_0.pdf</url></story><parent_chain><item><author>ted_dunning</author><text>This is yet another article that ignores the fact that there is a MUCH better approach to this problem.&lt;p&gt;Thompson sampling avoids the problems of multiple testing, power, early stopping and so on by starting with a proper Bayesian approach. The idea is that the question we want to answer is more &amp;quot;Which alternative is nearly as good as the best with pretty high probability?&amp;quot;. This is very different from the question being answered by a classical test of significance. Moreover, it would be good if we could answer the question partially by decreasing the number of times we sample options that are clearly worse than the best. What we want to solve is the multi-armed bandit problem, not the retrospective analysis of experimental results problem.&lt;p&gt;The really good news is that Thompson sampling is both much simpler than hypothesis testing can be done in far more complex situations. It is known to be an asymptotically optimal solution to the multi-armed bandit and often takes only a few lines of very simple code to implement.&lt;p&gt;See &lt;a href=&quot;http:&amp;#x2F;&amp;#x2F;tdunning.blogspot.com&amp;#x2F;2012&amp;#x2F;02&amp;#x2F;bayesian-bandits.html&quot; rel=&quot;nofollow&quot;&gt;http:&amp;#x2F;&amp;#x2F;tdunning.blogspot.com&amp;#x2F;2012&amp;#x2F;02&amp;#x2F;bayesian-bandits.html&lt;/a&gt; for an essay and see &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;tdunning&amp;#x2F;bandit-ranking&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;tdunning&amp;#x2F;bandit-ranking&lt;/a&gt; for an example applied to ranking.</text></item></parent_chain><comment><author>developer2</author><text>This ignores what I have seen in my experience, which is that marketing teams - composed of the people who dictate what A&amp;#x2F;B tests the business should run - have little to no background in statistics, let alone any interest whatsoever in actually performing legitimate A&amp;#x2F;B tests.&lt;p&gt;It&amp;#x27;s often the case that the decision maker has &lt;i&gt;already decided&lt;/i&gt; to move ahead with option A, but performs a minimal &amp;quot;fake&amp;quot; A&amp;#x2F;B test to put in their report as a way to justify their choice. I&amp;#x27;ve seen A&amp;#x2F;B tests deployed at 10am in the morning, and taken down at 1pm with less than a dozen data points collected. The A&amp;#x2F;B test &amp;quot;owner&amp;quot; is happy to see that option A resulted in 7 conversions, with option B only having 5. Not statistically significant whatsoever, but hey let&amp;#x27;s waste developers&amp;#x27; time and energy for two days implementing an A&amp;#x2F;B test in order to help someone else try to nab their quarterly marketing bonus.</text></comment>