<jupyter_start><jupyter_text>Machine Learning for Medicine: Workshop Variables and distributions OverviewData is a window into our patients.In medicine we care about two things: physiology and pathophysiology.Both of these translate to: how the body behaves, and how the body *mis*-behaves.Data is the way we try to figure out what's going on inside the patient and whether it's *normal* or *abnormal*.But we have to have a better understanding of where data comes from and what it can look like.In this notebook we cover some basics about variables, distributions, and data.These concepts are *core* to scientific and medical research, so we hope folks work through this before jumping into using machine learning in their dataset. How to run this notebookThis Jupyter notebook can be run simply by going to the menu up top, selecting 'Kernel -> Restart and Run All'.You'll need some basic libraries to get this to work.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import ipywidgets as widgets
from ipywidgets import interact, interactive
from IPython.display import clear_output, display, HTML<jupyter_output><empty_output><jupyter_text>What is a variable and a distributionLet's start right off with an example: I'm trying to figure out if my patient is diabetic.I could tell you what Mr. P's Hemoglobin A1c is right off the bat and you might be able to tell me whether Mr. P is above or below the ADA's threshold.That's not the point here.The point here is to understand what pieces are interacting as we make our inference.What we're interested in is two variables: the Hemoglobin A1c, which we'll label $x$, and whether Mr. P has diabetes, which we'll label $y$.$x$ and $y$ are *variables* we're interested in. A Healthy A1C: HistogramsBased on guidelines, we know that the A1c cutoff for diabetes is about X.We can think about how this cutoff was selected using a thought experiment.Some study was performed in healthy patients and patients with diabetes diagnosed through physical exam + urine findings.They plotted the A1c of the healthy patients as a *histogram*.What is a histogram?<jupyter_code>mfig,ax = plt.subplots()
mean = 5
xset = np.random.normal(5,2,size=1000)
histo = ax.hist(xset,bins=np.arange(0,40))
plt.xlim(0,40)
plt.ylim(0,500)<jupyter_output><empty_output><jupyter_text>**How do we read that?** A histogram is a way to display our data. The x-axis is the value of our A1c. The y-axis is the *number of patients that had that A1c value*.**The mean of healthy** We can find the mean A1c value in our healthy group pretty easily.<jupyter_code>mean_a1c = np.mean(xset)
print('The mean A1c is: ' + str(mean_a1c))<jupyter_output>The mean A1c is: 4.848157186947227<jupyter_text>Now let's take a look at the dataset we got from our *diabetes* patients. The diabetic A1cWe'll plot our diabetic dataset now in the same histogram format. The diabetic patients will have a *red* histogram while we'll keep our healthy patients in a *blue* histogram.<jupyter_code>mfig,ax = plt.subplots()
mean = 5
healthy_data = np.random.normal(5,2,size=1000)
histo = ax.hist(healthy_data,bins=np.arange(0,40))

diabetic_data = np.random.normal(6,2,size=1000)
histo = ax.hist(diabetic_data,bins=np.arange(0,40),color='red',alpha=0.7)
plt.xlim(0,40)
plt.ylim(0,500)<jupyter_output><empty_output><jupyter_text>So now we can take the mean of our *healthy* patients and the mean of our *diabetic* patients.<jupyter_code>healthy_mean_a1c = np.mean(healthy_data)
print('The mean A1c in our healthy patients is: ' + str(healthy_mean_a1c))

diabetic_mean_a1c = np.mean(diabetic_data)
print('The mean A1c in our diabetic patients is: ' + str(diabetic_mean_a1c))<jupyter_output>The mean A1c in our healthy patients is: 5.044257016318349
The mean A1c in our diabetic patients is: 5.896750148446977<jupyter_text>So, the means are **different**. But is this difference **meaningful**? Or is it just because measurements are always full of errors? After all, maybe we thought someone wasn't diabetic when they actually were.This is when we work with *statistical tests*.  Putting it all together with an interactive example<jupyter_code>hm=20
hv=5
hset = np.random.normal(hm,hv,size=1000)


#Separate Scripts
def update_distr(cutoff):
    plt.figure()
    #fig = plt.figure()
    #ax = fig.add_axes([0, 0, 1, 1])
    #ax.axis('off')
    
    #mfig,ax = plt.subplots()
    #ax.cla()
    histo = plt.hist(hset,bins=np.arange(0,40))
    line = plt.axvline(x=cutoff,color='red')
    #Fix up our plotting here
    plt.xlim(0,40)
    plt.ylim(0,500)
    
    
    # Do some basic stats here
    pval = np.sum(hset > cutoff) / 1000
    plt.title('P-val: ' + str(pval))
    
    plt.show()


w = interactive(update_distr, cutoff=(0.,40.))
display(w)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08752093802345058, 'fraction_numerical': 0.029731993299832497, 'mean_word_length': 4.956359102244389, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '410255', 'license': 'MIT', 'n_tokens_mistral': 1639, 'n_tokens_neox': 1484, 'n_words': 597, 'path': 'intermediate/.ipynb_checkpoints/ML_Med_distributions-checkpoint.ipynb', 'repo_name': 'virati/Med_ML', 'size': '4776'}