<commit_before>---
layout: post
title: Linear Regression Analysis - Part 1
---

Let's talk about correlation.

General correlation is the quantification of a relationship between variables, most commonly represented as X & Y.
The goal of correlation is to identify a factor, X, that we can use to make useful predictions about Y. We'll get into the
philisophical arguments about extrapolation vs interpolation at another time, but for now, it's just something to mention. 

Ideally, we want to make causative connections between variables in data sets, but in the real world this is difficult due to the
wide array of interactions that truly exist. You often hear, "Correlation does not imply causation" thrown around. While this is 
true, it can be misleading. It does not imply causation, but it also doesn't negate it. In general, if you find high correlation,
you should continue whatever reasoning led you to that point, but refine your methods and ask further related questions.

Should you find correlation, you'll be presented with a number. That number will tell you a lot. It will tell you whether
the relationship is positive, with a upward sloping line, or a negative one, with a downward slope. It will tell you how strong 
the relationship is, a higher number implying a more useful predictive quality. 

Let's say you find a correlation of 0.75, this simply means that 75% of your Y's behaviour can be predicted by variable X. 
We'd all like to be able to predict future outcomes, and linear correlation is one way we can. It's not without risks if you dump resources into a faulty correlation, but when done correctly, the rewards can be worth the investment. 
<commit_msg>Update 2017-10-28-LinearReg.md<commit_after>---
layout: post
title: Linear Regression Analysis - Part 1
---

Let's talk about correlation.

General correlation is the quantification of a relationship between variables, most commonly represented as X & Y.
The goal of correlation is to identify a factor, X, that we can use to make useful predictions about Y. We'll get into the
philisophical arguments about extrapolation vs interpolation at another time, but for now, it's just something to mention. 

Ideally, we want to make causative connections between variables in data sets, but in the real world this is difficult due to the wide array of interactions that truly exist. You often hear, "Correlation does not imply causation" thrown around. While this is true, it can be misleading. It does not imply causation, but it also doesn't negate it. In general, if you find high correlation, you should continue whatever reasoning led you to that point, but refine your methods and ask further related questions.

Should you find correlation, you'll be presented with a number ranging for -1 to +1. That number will tell you a lot. It will tell you whether the relationship is positive, with a upward sloping line, or a negative one, with a downward slope. It will tell you how strong the relationship is, a higher number implying a more useful predictive quality. The accepted threshold
for a correlation coefficient that you should take seriously, is less than -0.5 or greater than +0.5.

Let's say you find a correlation of 0.75, this simply means that 75% of your Y's behaviour can be predicted by variable X. 
We'd all like to be able to predict future outcomes, and linear correlation is one way we can. It's not without risks if you dump resources into a faulty correlation, but when done correctly, the rewards can be worth the investment. 

{'fraction_non_alphanumeric': 0.0395703787450537, 'fraction_numerical': 0.007348784624081402, 'mean_word_length': 4.354009077155824, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2620984', 'n_tokens_mistral': 857, 'n_tokens_neox': 790, 'n_words': 578}