<commit_before>---
layout: post
title: Deceitful Data
categories:
- Thoughts
---

Let's pretend for a second that you're a rogue statistician with nothing to lose. Maybe you were a vinyl record company's top stats guy until your devil-may-care graphing methods got someone hurt, and they let you go. Maybe you wanted to get back at them for that. Putting aside your once-valued code of ethics (because what good has ethics ever done?), you spend a few minutes gathering some data and begin.

<img src="/assets/images/38.png">
<p>Source: Vinyl - <a href="http://www.theregister.co.uk/2013/04/10/ifpi_annual_music/" target="_blank">IFPI yearbook</a>, Temperature - <a href="http://www.ncdc.noaa.gov/temp-and-precip/time-series/index.php?parameter=tmp&month=12&year=2012&filter=12&state=110&div=0" target="_blank">NOAA data center</a>.</p>

And there you have it, a statistical correlation based on real data suggesting vinyl sales are causing global warming. A correlation like this is completely meaningless, but it'll get the newspapers going. And you're not done. It's not enough that vinyl must fail, its competitors must succeed. Vinyl competes with digital music. You decide to combine that with the irrefutable knowledge that everyone likes kittens.

<img src="/assets/images/39.png">
<p>Source: Digital Music - <a href="http://www.aria.com.au/documents/MediaRelease-ARIAannounce2012wholesalefigures.pdf" target="_blank">ARIA figures</a>, Kittens - <a href="http://www.google.com/trends/explore?hl=en#q=cat&date=4%2F2009%2049m&cmpt=q" target="_blank">Google Trends</a>.</p>

You decide that you're having so much fun with this that you branch out a bit. You've always liked that chap in the whitehouse, so you want to help him out a bit. Maybe if more people looked up to him as a hero. As they would a Man of Steel, perhaps?

<img src="/assets/images/40.png">
<p>Source: Obama - <a href="http://www.presidency.ucsb.edu/data/popularity.php" target="_blank">UCSB</a>, Superman - <a href="http://www.google.com/trends/explore?hl=en#q=superman&date=11%2F2012%202m&cmpt=q" target="_blank">Google Trends</a>.</p>

Feeling satisfied with your work, you decide to get a victory coffee in celebration. You go down to your local cafe but Bentley, the owner, tells you that they've increased their prices by five cents. You should have known not to trust anyone named <em>Bentley</em>. In fact...

<img src="/assets/images/44.png">
<p>Source: Bentley - <a href="http://nametrends.net/name.php?name=Bentley" target="_blank">Name Trends</a>, Coffee Prices - <a href="http://www.gilkatho.com.au/August-2012-Gilkatho-cappuccino-Price-Index" target="_blank">Gilkatho</a>.</p>

You knew it!

## Correlated Lies
My point is, of course, to treat correlations with a grain of salt. In light of the media's coverage of recent tragedies it's good to remind ourselves that even if there's an apparent link between the incident rates of two events, its nothing more than coincidence without a causal link to go with it. Any statistician can manipulate data to tell the story that they want to tell. None of the data used here has been altered, just framed in a way to 'prove' a certain hypothesis. Numbers may not lie, but people do.<commit_msg>Removed links causing build failure<commit_after>---
layout: post
title: Deceitful Data
categories:
- Thoughts
---

Let's pretend for a second that you're a rogue statistician with nothing to lose. Maybe you were a vinyl record company's top stats guy until your devil-may-care graphing methods got someone hurt, and they let you go. Maybe you wanted to get back at them for that. Putting aside your once-valued code of ethics (because what good has ethics ever done?), you spend a few minutes gathering some data and begin.

<img src="/assets/images/38.png">

And there you have it, a statistical correlation based on real data suggesting vinyl sales are causing global warming. A correlation like this is completely meaningless, but it'll get the newspapers going. And you're not done. It's not enough that vinyl must fail, its competitors must succeed. Vinyl competes with digital music. You decide to combine that with the irrefutable knowledge that everyone likes kittens.

<img src="/assets/images/39.png">

You decide that you're having so much fun with this that you branch out a bit. You've always liked that chap in the whitehouse, so you want to help him out a bit. Maybe if more people looked up to him as a hero. As they would a Man of Steel, perhaps?

<img src="/assets/images/40.png">

Feeling satisfied with your work, you decide to get a victory coffee in celebration. You go down to your local cafe but Bentley, the owner, tells you that they've increased their prices by five cents. You should have known not to trust anyone named <em>Bentley</em>. In fact...

<img src="/assets/images/44.png">

You knew it!

## Correlated Lies
My point is, of course, to treat correlations with a grain of salt. In light of the media's coverage of recent tragedies it's good to remind ourselves that even if there's an apparent link between the incident rates of two events, its nothing more than coincidence without a causal link to go with it. Any statistician can manipulate data to tell the story that they want to tell. None of the data used here has been altered, just framed in a way to 'prove' a certain hypothesis. Numbers may not lie, but people do.
{'fraction_non_alphanumeric': 0.08738404452690167, 'fraction_numerical': 0.011873840445269016, 'mean_word_length': 4.529230769230769, 'pattern_counts': {'":': 0, '<': 39, '<?xml version=': 0, '>': 39, 'https://': 0, 'lorem ipsum': 0, 'www.': 7, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1347071', 'n_tokens_mistral': 1586, 'n_tokens_neox': 1465, 'n_words': 748}