<story><title>A/B testing gets misused to juice metrics in the short term</title><url>https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-destroys-everything-it-touches/</url></story><parent_chain><item><author>causi</author><text>The problem with AB testing is that it&amp;#x27;s a short-term strategy. For example, if a news site runs AB testing with headlines, they&amp;#x27;ll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles get the most alluring headlines.</text></item></parent_chain><comment><author>tomrod</author><text>This highlights the major downside to &amp;quot;data-driven&amp;quot; policy and decisions.&lt;p&gt;Data can &amp;quot;lie&amp;quot;. What is observed is not always reality, simply what we can see of it.&lt;p&gt;Consider auctions. You never actually &amp;quot;see&amp;quot; the bidder&amp;#x27;s demand or utility. Yes, there are some ways to structure auctions that in theory show willingness to pay and such (ignoring confounding factors and irrationality), but you don&amp;#x27;t actually observe anything beyond the bid.&lt;p&gt;Similarly, on websites, you don&amp;#x27;t always know the causal reasons people click here or there. You know perhaps enough to predict a step-wise behavior, but don&amp;#x27;t (usually) understand the full behavioral lifecycle -- especially if a metric improves but at the hidden cost of decrements to conversion and similar.</text></comment>
<story><title>A/B testing gets misused to juice metrics in the short term</title><url>https://www.zumsteg.net/2022/07/05/unchecked-ab-testing-destroys-everything-it-touches/</url></story><parent_chain><item><author>causi</author><text>The problem with AB testing is that it&amp;#x27;s a short-term strategy. For example, if a news site runs AB testing with headlines, they&amp;#x27;ll find that bullshit clickbait headlines get more pageviews than concise, accurate headlines, but the constant use of clickbait headlines will over time destroy overall traffic to your site. More frustratingly, sites run by smart people tend to fall into a balance where the worst articles get the most alluring headlines.</text></item></parent_chain><comment><author>goodside</author><text>There’s nothing about AB testing that requires you to use short-term metrics. I used to manage AB tests for online dating sites (OkCupid, Grindr) where subscription revenue is what matters, and the gains of any strategy will take months to materialize. We were well aware that, say, raising prices would yield more short-term revenue at the expense of long-term revenue. That didn’t stop us from testing, it just made the statistics more complicated.</text></comment>