<story><title>37signals: A/B testing part 3: Finalé</title><url>http://37signals.com/svn/posts/2991-behind-the-scenes-ab-testing-part-3-final</url></story><parent_chain><item><author>aresant</author><text>I would love to see the data sets that support the lifts - those little &quot;up 3.49 / down 3.38&quot; can get notoriously tricky in terms of statistical relevancy using GWO or otherwise.&lt;p&gt;Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns.&lt;p&gt;Sometimes the sheer &quot;newness&quot; can skew the data as folks sitting on the fence (returning visitors) push the convo data up.&lt;p&gt;All that said, I&apos;m a fan of the new look and feel, fun to watch 37signals influence turn the market - expect to see lots of similar looking stuff in the future.</text></item></parent_chain><comment><author>sunir</author><text>There are a couple important biases where just changing something results in a positive result.&lt;p&gt;* &lt;a href=&quot;http://en.wikipedia.org/wiki/Novelty_effect&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Novelty_effect&lt;/a&gt; (everyone loves seeing something new!)
* &lt;a href=&quot;http://en.wikipedia.org/wiki/Hawthorne_effect&quot; rel=&quot;nofollow&quot;&gt;http://en.wikipedia.org/wiki/Hawthorne_effect&lt;/a&gt; (being a subject of an experiment is motivational)&lt;p&gt;There are a couple ways to deal with these biases. One is to try a large number of different designs, so the bias is spread across a large number of options. Thus the comparative winner is more likely to itself be intrinsically better, at least amongst the new designs. Note that this still negatively punishes the control.&lt;p&gt;The other way is to wait longer until the new design is no longer &apos;new&apos;. In practice, that is impractical! Really, all you can do is pick the winner and keep monitoring it after the fact to see if there is a regression in the result.&lt;p&gt;I know others have said this, but you can harness these biases. Simply change the design frequently for no reason other than to stay &apos;fresh&apos;. It keeps people interested and it shows you&apos;re alive and kicking.</text></comment>
<story><title>37signals: A/B testing part 3: Finalé</title><url>http://37signals.com/svn/posts/2991-behind-the-scenes-ab-testing-part-3-final</url></story><parent_chain><item><author>aresant</author><text>I would love to see the data sets that support the lifts - those little &quot;up 3.49 / down 3.38&quot; can get notoriously tricky in terms of statistical relevancy using GWO or otherwise.&lt;p&gt;Also to note, I often find that radical new variant designs for well established clients have rapidly diminishing returns.&lt;p&gt;Sometimes the sheer &quot;newness&quot; can skew the data as folks sitting on the fence (returning visitors) push the convo data up.&lt;p&gt;All that said, I&apos;m a fan of the new look and feel, fun to watch 37signals influence turn the market - expect to see lots of similar looking stuff in the future.</text></item></parent_chain><comment><author>chc</author><text>The article actually notes that those up/down 2% results are statistically insignificant (&quot;a specific person didn’t quite matter among the set of people we tested&quot;). But the lift from no-person to big-smiling-face was very significant: more than 100%.</text></comment>