<jupyter_start><jupyter_text>Describing distributionsWe have seen several examples of *distributions*.We can describe distributions as having a *center*, and a *spread*.In [the mean as predictor](mean_meaning), we saw that the mean isa useful measure of the center of a distribution.What measure should we use for the spread?  Back to chronic kidney diseaseWe return to the [data on chronic kidney disease]({{ site.baseurl}}/data/chronic_kidney_disease).Download the data to your computer via this link: [ckd_clean.csv]({{site.baseurl }}/data/ckd_clean.csv).<jupyter_code>ckd_full = pd.read_csv('ckd_clean.csv')
ckd_full.head()<jupyter_output><empty_output><jupyter_text>We will use this dataset to get a couple of variables (columns) andtherefore a couple of distributions.Let's start with the White Blood Cell Count, usually abbreviated as WBC.<jupyter_code>wbc = ckd_full['White Blood Cell Count']
wbc.hist()
plt.title('White Blood Cell Count');
wbc.describe()<jupyter_output><empty_output><jupyter_text>Compare this to Hemoglobin concentrations:<jupyter_code>hgb = ckd_full['Hemoglobin']
hgb.hist()
plt.title('Hemoglobin');
hgb.describe()<jupyter_output><empty_output><jupyter_text>Notice that we can't easily plot these two on the same axes, becausetheir units are so different. Here's what that looks like.  Notice that the hemoglobin values disappear in a tiny spike to the left.<jupyter_code># Use alpha to make the histograms a little transparent.
# Label them for a legend.
hgb.hist(alpha=0.7, label='HGB')
wbc.hist(alpha=0.7, label='WBC')
plt.title("HGB and WBC together - HGB tiny spike at left")
plt.legend();<jupyter_output><empty_output><jupyter_text>We could try and fix this by subtracting the mean, as a center value, sothe values are now *deviations* from the mean.<jupyter_code>wbc_deviations = wbc - np.mean(wbc)
wbc_deviations.hist()
plt.title('White Blood Cell Count deviations');
hgb_deviations = hgb - np.mean(hgb)
hgb_deviations.hist()
plt.title('Hemoglobin deviations');<jupyter_output><empty_output><jupyter_text>The deviations each have a mean very very close to zero, and therefore,they have the same center:<jupyter_code>np.mean(wbc_deviations), np.mean(hgb_deviations)<jupyter_output><empty_output><jupyter_text>We still cannot sensibly plot them on the same axes, because the WBC values havea very different *spread*.  The WBC values completely dominate the x axis ofthe graph.  We can't reasonably compare the WBC deviations to theHemoglobin deviations, because they have such different *units*.<jupyter_code>hgb_deviations.hist(alpha=0.7, label='HGB')
wbc_deviations.hist(alpha=0.7, label='WBC')
plt.title("HGB and WBC deviations - you can't see HGB")
plt.legend();<jupyter_output><empty_output><jupyter_text>We would like a measure of the spread of the distribution, so we can setthe two distributions to have the same spread.  The standard deviation In the [mean as predictor](mean_meaning) section, we found that mean wasthe best value to use as a predictor, to minimize the sum of *squared*deviations.Maybe we could get an idea of the typical *squared* deviation, asa measure of spread?<jupyter_code>hgb_deviations[:10]
hgb_dev_sq = hgb_deviations ** 2
hgb_dev_sq[:10]
hgb_dev_sq.hist()
plt.title('HGB squared deviations')<jupyter_output><empty_output><jupyter_text>The center, or typical value, of this distribution, could be the *mean*.<jupyter_code>hgb_dev_sq_mean = np.mean(hgb_dev_sq)
hgb_dev_sq_mean<jupyter_output><empty_output><jupyter_text>This is the *mean squared deviation*.  This is also called the*variance*.  Numpy has a function to calculate that in one shot:<jupyter_code># The mean squared deviation is the variance
np.var(hgb)<jupyter_output><empty_output><jupyter_text>The mean squared deviation is a good indicator of the typical squareddeviation.  What should we use for some measure of the typicaldeviation?We could take the square root of the mean squared deviation, like this:<jupyter_code>np.sqrt(hgb_dev_sq_mean)<jupyter_output><empty_output><jupyter_text>This is a measure of the spread of the distribution.  It is a measure ofthe typical or average deviation.It is also called the *standard deviation*.<jupyter_code>np.std(hgb)<jupyter_output><empty_output><jupyter_text>We can make our distribution have a standard center *and* a standardspread by dividing our mean-centered distribution, by the standarddeviation.  Then the distribution will have a standard deviation veryclose to 1.This version of the distribution, with mean 0 and standard deviation of1, is called the *standardized* distribution.<jupyter_code>standardized_hgb = hgb_deviations / np.std(hgb)
standardized_hgb.hist()
plt.title('Standardized Hemoglobin')<jupyter_output><empty_output><jupyter_text>We can make a function to do this:<jupyter_code>def standard_units(x):
    return (x - np.mean(x))/np.std(x)
std_hgb_again = standard_units(hgb)
std_hgb_again.hist()
plt.title('Standardized Hemoglobin, again')<jupyter_output><empty_output><jupyter_text>If we do the same to the WBC, we can compare values of thedistributions:<jupyter_code>std_wbc = standard_units(wbc)
std_wbc.hist()
plt.title('Standardized White Blood Cell Count')<jupyter_output><empty_output><jupyter_text>Now we can put both these distributions on the same graph, to compare them directly.<jupyter_code>std_hgb_again.hist(alpha=0.7, label='HGB')
std_wbc.hist(alpha=0.7, label='WBC')
plt.title('Standardized HGB and WBC')
plt.legend()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09197293837995978, 'fraction_numerical': 0.0036569756811117207, 'mean_word_length': 6.950581395348837, 'pattern_counts': {'":': 0, '<': 65, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '289426', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 1863, 'n_tokens_neox': 1723, 'n_words': 562, 'path': 'notebooks/08/standard_scores.ipynb', 'repo_name': 'matthew-brett/cfd-uob', 'size': '5469'}