<jupyter_start><jupyter_text>Regression to the mean Students take a standard exam at two time points. Getting two grades. If the 1'st grade is higher than the average, the 2'nd grade will tend to be lower than the 1'st. If the 1'st grade is lower than the average, the 2'nd grade will tend to be higher than the 1'st. This happens even if the two grades are **statistically independent**. This is called **regression to the mean**<jupyter_code>%pylab inline
import pandas as pd
import numpy as np
from numpy import arange,array,ones,linalg<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>This notebook is a an intuitive introduction to the concept of linear regression. Partially based on Chapter 10 in the book [**Statistics** by Freedman, Pisani and Purves](https://www.amazon.com/Statistics-4th-David-Freedman/dp/0393929728)   Fathers and SonsA classical example of regression to the mean involves the height of fathers and sons. We take our data from here http://www.math.uah.edu/stat/data/Pearson.html<jupyter_code>HH=pd.read_csv('data/Pearson.csv')
HH.head()
# Copute some basic descritive statistics.
HH.describe()
A=np.array(HH['Father'])
A=np.array([ones(len(A)),A])
y=np.array(HH['Son'])
A.shape,y.shape

w1 = linalg.lstsq(A.T,y)[0] # finding the optimal parameters
print("The regression function from father's height to to son's height is")
print("son's height = %f + %f * father's height"%(w1[0],w1[1]))
def f(x,w):
    return w[0]+w[1]*x
ax=HH.plot(kind='scatter',s=1,x='Father',y='Son',figsize=[6,5]);
x0,x1=xlim()
ax.plot([x0,x1],[f(x0,w1),f(x1,w1)],'r');
title("Regression of Father's height to Son's height");<jupyter_output><empty_output><jupyter_text>Note that as father's height varies from 58" to 77" the son's height varies only from 63" to 72"<jupyter_code>HH['Son-Father']=HH['Son']-HH['Father']
A=np.array(HH['Father'])
A=np.array([ones(len(A)),A])
y=np.array(HH['Son-Father'])
A.shape,y.shape

w2 = linalg.lstsq(A.T,y)[0] # finding the optimal parameters
w2<jupyter_output>C:\Users\bishw\Anaconda3\lib\site-packages\ipykernel_launcher.py:6: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.<jupyter_text>Consider the differenceRegression to the mean is even more evident when we compare the height of the father to the **difference** between the heights of the son and the father.<jupyter_code>ax=HH.plot(kind='scatter',s=1,x='Father',y='Son-Father',figsize=[6,5]);
x0,x1=xlim()
ax.plot([x0,x1],[f(x0,w2),f(x1,w2)],'r');
grid()
title("Regression of Father's height to the difference between Son and Father");<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13561113067981684, 'fraction_numerical': 0.022190912293060937, 'mean_word_length': 6.282051282051282, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '399807', 'license': 'MIT', 'n_tokens_mistral': 1018, 'n_tokens_neox': 956, 'n_words': 308, 'path': 'Week 12 _ Regression and PCA/lectures/2.Regressing to the mean.ipynb', 'repo_name': 'ebishwaraj/ProbabilityStatisticsPython_DSE210x_UCSD_DataScienceMicroMasters', 'size': '2839'}