<story><title>Winning A/B results were not translating into improved user acquisition</title><url>http://blog.sumall.com/journal/optimizely-got-me-fired.html</url></story><parent_chain></parent_chain><comment><author>pmiller2</author><text>The red flag here for me was that Optimizely encourages you to stop the test as soon as it &amp;quot;reaches significance.&amp;quot;  You shouldn&amp;#x27;t do that.  What you should do is precalculate a sample size based on the statistical power you need, which involves determining your tolerance for the probability of making an error and on the minimum effect size you need to detect.  Then, you run the test to completion and crunch the numbers afterward.  This helps prevent the scenario where your page tests 18% better than itself by minimizing probability that your &amp;quot;results&amp;quot; are just a consequence of a streak of positive results in one branch of the test.&lt;p&gt;I was also disturbed that the effect size was taken into account in the sample size selection.  You need to know this before you do any type of statistical test.  Otherwise, you are likely to get &amp;quot;positive&amp;quot; results that just don&amp;#x27;t mean anything.&lt;p&gt;OTOH, I wasn&amp;#x27;t too concerned that the test was a one-tailed test.  Honestly, in a website A&amp;#x2F;B test, all I really am concerned about is whether my new page is better than the old page.  A one-tailed test tells you that.  It might be interesting to run two-tailed tests just so you can get an idea what not to do, but for this use I think a one-tailed test is fine.  It&amp;#x27;s not like you&amp;#x27;re testing drugs, where finding any effect, either positive or negative, can be valuable.&lt;p&gt;I should also note that I only really know enough about statistics to not shoot myself in the foot in a big, obvious way.  You should get a real stats person to work on this stuff if your livelihood depends on it.</text></comment>
<story><title>Winning A/B results were not translating into improved user acquisition</title><url>http://blog.sumall.com/journal/optimizely-got-me-fired.html</url></story><parent_chain></parent_chain><comment><author>josefresco</author><text>This article comes off as a bit boastful and somewhat of an advertisement for the company...&lt;p&gt;&amp;quot;What threw a wrench into the works was that SumAll isn’t your typical company. We’re a group of incredibly technical people, with many data analysts and statisticians on staff. We have to be, as our company specializes in aggregating and analyzing business data.  Flashy, impressive numbers aren’t enough to convince us that the lifts we were seeing were real unless we examined them under the cold, hard light of our key business metrics.&amp;quot;&lt;p&gt;I was expecting some admission of how their business is actually different&amp;#x2F;unusual, not just &amp;quot;incredibly technical&amp;quot;.  Secondly, I was expecting to hear that these &amp;quot;technical&amp;quot; people monkeyed with the A&amp;#x2F;B testing (or simply over-thought it) which got them in to trouble .. but no, just a statement about how &amp;quot;flashy&amp;quot; numbers don&amp;#x27;t appeal to them.&lt;p&gt;I think the article would be much better without some of that background.</text></comment>