<story><title>The Onion cut 66% of their bandwidth by upstream caching 404s</title><url>http://www.reddit.com/r/django/comments/bhvhz/the_onion_uses_django_and_why_it_matters_to_us/c0mvow7</url></story><parent_chain><item><author>randomstring</author><text>OK, so what they are saying is that The Onion is throwing away 66% of its traffic by serving up 404s instead of serving up page views. Wouldn&apos;t they  1) have a better user experience by redirecting to the correct URL 2) have more page views and most important 3) make more money off of ads?&lt;p&gt;The Internet is trying to give the gift of free traffic and The Onion is saying: &quot;no thanks.&quot; Most sites have to pay for traffic, and wouldn&apos;t be trowing it away.&lt;p&gt;Way to play it like Big Media. What&apos;s next? A pay wall?</text></item></parent_chain><comment><author>lmkg</author><text>66% of server time, not 66% of traffic. I am not a web developer, but my guess of what&apos;s happening: active pages get cached, so 500,000 page views to the homepage uses as much server time as a spider coming across a dead link from 2002. The Onion updates frequently, but not hundreds of time a day, and there&apos;s only a (relatively) small number of &apos;current&apos; articles at any given time. It wouldn&apos;t take that many dead links to dwarf the server processing time, as long as they were all distinct links.&lt;p&gt;As a web analyst, I can tell you that in general old, deprecated content does not get very many visits except from spiders. I would not at all be surprised if the marginal ad revenue is break-even compared to the extra server load.&lt;p&gt;Note also that they&apos;re not just throwing the traffic away. It&apos;s a decent 404. It&apos;s not the best I&apos;ve seen, there&apos;s room for improvements, but it&apos;s directing people straight to the archive so they can look for the story they were linked to.</text></comment>
<story><title>The Onion cut 66% of their bandwidth by upstream caching 404s</title><url>http://www.reddit.com/r/django/comments/bhvhz/the_onion_uses_django_and_why_it_matters_to_us/c0mvow7</url></story><parent_chain><item><author>randomstring</author><text>OK, so what they are saying is that The Onion is throwing away 66% of its traffic by serving up 404s instead of serving up page views. Wouldn&apos;t they  1) have a better user experience by redirecting to the correct URL 2) have more page views and most important 3) make more money off of ads?&lt;p&gt;The Internet is trying to give the gift of free traffic and The Onion is saying: &quot;no thanks.&quot; Most sites have to pay for traffic, and wouldn&apos;t be trowing it away.&lt;p&gt;Way to play it like Big Media. What&apos;s next? A pay wall?</text></item></parent_chain><comment><author>ShabbyDoo</author><text>&quot;The Onion is throwing away 66% of its traffic by serving up 404s instead of serving up page views.&quot;&lt;p&gt;Spidering and organic activity are bundled together in this 66% number.  It would be interesting to know what percentage of this was people who wanted to see an Onion page.</text></comment>