<story><title>p-Hacking and False Discovery in A/B Testing</title><url>https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791</url></story><parent_chain></parent_chain><comment><author>boron1006</author><text>I would be shocked if it were as low as 57%. As an intern, I found that the analysts in charge of A&amp;#x2F;B tests often didn&amp;#x27;t have a background in science or running experiments, and didn&amp;#x27;t really care. There were a couple of data analytics teams in the company, and I think a lot of the developers didn&amp;#x27;t like my team because we were seen as &amp;quot;fussier&amp;quot; than the other one. We required people to preregister hypotheses, and run experiments for predetermined amounts of time.&lt;p&gt;I don&amp;#x27;t think the tech environment is very conducive to running experiments. Everything moves too fast, by the time you figure out the results someone gives you are bs, they&amp;#x27;ve already got promoted 3 times and work as a director at a different company.&lt;p&gt;I work in science now, and although people still p-hack like hell, there&amp;#x27;s at least some sort of shame about it. There&amp;#x27;s a long term cost too, I&amp;#x27;ve met a couple researchers who have spent years trying to replicate some finding they got early in their career through suspicious means.</text></comment>
<story><title>p-Hacking and False Discovery in A/B Testing</title><url>https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3204791</url></story><parent_chain></parent_chain><comment><author>aisscott</author><text>Hi, I am one of the authors. We found that people p-hack with traditional t-tests. Most A&amp;#x2F;B tests were run this way in the past and some still are. The paper is using Optimizely data (from 2014) before Optimizely introduced new testing in 2015 designed to solve the issues we found in the paper.&lt;p&gt;If you want to know how Optimizely prevents p-hacking check out the math behind Optimizely’s current testing here: &lt;a href=&quot;https:&amp;#x2F;&amp;#x2F;www.optimizely.com&amp;#x2F;resources&amp;#x2F;stats-engine-whitepaper&amp;#x2F;&quot; rel=&quot;nofollow&quot;&gt;https:&amp;#x2F;&amp;#x2F;www.optimizely.com&amp;#x2F;resources&amp;#x2F;stats-engine-whitepaper...&lt;/a&gt;</text></comment>