<jupyter_start><jupyter_text>Machine learning for medicine Understanding Correlation OverviewIn medicine, we care about how differents physiologic processes *relate* to each other.Correlation is one way to measure how related two things are.In this notebook we get hands on with *linear* correlation.  Code Setup<jupyter_code>import numpy as np
import scipy

from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import scipy.stats as stats
from example_systems import *

import matplotlib
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(style="darkgrid")
matplotlib.rcParams['figure.figsize'] = [15, 10]<jupyter_output><empty_output><jupyter_text>What is correlationCorrelations are the backbone of science.Correlations are one way to assess whether two variables are *related* to each other.Correlation checks to see whether there's a *linear* relationship between two variables $X$ and $Y$.In order words: if we *double* $X$ do we double $Y$?For some things, this is reasonable.<jupyter_code>interact(simple_eg,slope=(-5,5,0.1),noise=(0.0,10.0,0.5),samples=fixed(100));<jupyter_output><empty_output><jupyter_text>We can play around with these two variables (blue and red).The red variable *isn't* linearly correlated with x.No matter what we set x at, negative, zero, positive, the value of y is 0.The blue variable *is* linearly correlated with x.The slope of the relationship tells us a bit about how *robust* the relationship is.It doesn't mean much in the absence of noise.Now, start adding some *noise*.The key thing here is that noise doesn't change the underlying relationship between our variables.It may *look* like it does, but that's just because other things are interfering with what we care about.Imagine static over the radio, or buffering in your music stream.  Limited sample sizeLet's do the same sort of analysis, but change the number of samples we have available to us.<jupyter_code>interact(simple_eg,slope=fixed(0.2),noise=fixed(2.0),samples=(2,50,1));
interact(simple_eg,slope=fixed(0.2),noise=(0.0,10.0,0.1),samples=(2,50,1));<jupyter_output><empty_output><jupyter_text>You can think about this like the number of samples of a patient's Try sliding the samples down from 50 to 2 and see what happens to the p-value.  Signal-to-NoiseThe last point we'll make is an important one that applies to everything in medicine (and science).The idea of a *signal-to-noise* ratio.We can see this by looking at the previous example: we can control the *slope* and the *noise*.Turns out the slope is what we're interested in: that's the **signal** that we're trying to understand: are these two variables related and *how much* are they related.Noise, on the other hand, comes in and messes things up indiscriminantly.It's important to acknowledge that when we're focusing on just two variables to see if they're related, the 'slope' doesn't matter that much.Where it comes into play is when we consider the two variables (in the context of other variables*.Then we have to start asking questions about how variables affect each other *relative* to other variables.<jupyter_code>interact(simple_eg,slope=(0,5,0.01),noise=(0.0,10.0,0.1),samples=fixed(20));<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0798035604665439, 'fraction_numerical': 0.017495395948434623, 'mean_word_length': 5.8900634249471455, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '199676', 'license': 'MIT', 'n_tokens_mistral': 962, 'n_tokens_neox': 904, 'n_words': 414, 'path': 'intermediate/ML_Med_corr.ipynb', 'repo_name': 'virati/Med_ML', 'size': '3258'}