<jupyter_start><jupyter_text>Day 2: Basics of Statistics and Basics of Machine Learning- Random Variables- Common Distributions- Linear Regression- Logistic Regression<jupyter_code>import pandas as pd
import numpy as np
import statsmodels.formula.api as smf

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>Day 2: Basics of Machine Learning Linear Regression Loading and inspecting the dataWe will use a dataset about various species of insects (Source: CrossValidated).  We will be interested in explaining how the insects wing span varies, as influenced by the other measurements in the dataset.Our first step is to take a quick look at the raw data.<jupyter_code>!head ./data/insects.csv<jupyter_output>continent	latitude	wingsize	sex
1	40.5	941.111111111	0
1	42.0	924.444444444	0
1	43.6	924.204444444	0
1	45.7	915.217777778	0
1	45.9	905.471111111	0
1	47.4	896.004444444	0
1	50.0	913.0	0
1	51.8	916.44	0
1	53.8	933.417777778	0<jupyter_text>**Discussion:** How would you describe this dataset?  How many variables are there?  How would you describe these variables?It looks like there are four columns in our dataset: `continent`, `latitude`, `wingsize`, and `sex`.Elements in rows are separated from each other using a tab character; this type of format is often called "tab-separated data".<jupyter_code>insects_df = pd.read_csv('./data/insects.csv', sep='\t')<jupyter_output><empty_output><jupyter_text>Now we have the data in a Python object:<jupyter_code>insects_df.head()<jupyter_output><empty_output><jupyter_text>We've got our four columns `continent`, `latitude`, `wingsize`, and `sex`.We can see some short descriptions of their qualities using `info`:<jupyter_code>insects_df.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 42 entries, 0 to 41
Data columns (total 4 columns):
continent    42 non-null int64
latitude     42 non-null float64
wingsize     42 non-null float64
sex          42 non-null int64
dtypes: float64(2), int64(2)
memory usage: 1.4 KB<jupyter_text>Looking at the dataWe can get a first feel for how the quantities in our data are spread out using **histograms**:<jupyter_code>fig, axs = plt.subplots(2, 2, figsize=(10, 8))

for ax, column in zip(axs.flatten(), insects_df.columns):
    ax.hist(insects_df[column])
    ax.set_title(column)

fig.tight_layout()<jupyter_output><empty_output><jupyter_text>**Discussion:** What did you learn from these histograms?  How do they help you describe the data?Some observations:- `continent` and `sex` take only two values.  There are two continents represented in the data, labeled zero and one as well as two sexes (probably Male and Female), also labeled zero and one.These zero/one columns are called **binary** or **indicator variables**, they measure a specific yes/no condition.- The values of `wingspan` cluster into two distinct groups.  This is very interesting and worthy of further investigation. ScatterplotsHistograms are useful, but limited, as they do not reveal anything about the *relationships between the columns in our data*.  Insread, we will turn to undoubtedly  the most effective and flexible visualization: the **scatterplot**.<jupyter_code>_, ax = plt.subplots()

ax.scatter(insects_df.latitude, insects_df.wingsize, s=40)
ax.set_xlabel('Latitude')
ax.set_ylabel('Wing Size')
ax.set_title('Insect wing sizes at various latitudes')<jupyter_output><empty_output><jupyter_text>Here we have a scatterplot of `wingsize` against `latitude`.**Discussion:** What patterns do you see in the scatterplot.  Can you form any hypothesis about the data?Here are some thoughts:- The most prominent feature of this data is the two bands.  There seem to be two very well defined elongated clusters of data, with the average wing size in one cluster much greater than in the other.- Within each cluster there is noticeable tendency for wing size to first decrease and then increase as latitude varies.This leads to a few questions we may wish to answer with the data.1. Are the two clusters associated with one of the other two variables in the dataset, continent or sex?2. Is the increase of wing size as latitude increases real or illusory?Let's answer each of these questions. Are the two clusters associated with either continent or sex?We can discover if the two clusters in the data are associated with either `continent` or `sex` through a well chosen visualization.  Let's make the same scatterplot from before, but color each point either red or blue, according to the value of `continent` or `sex`.<jupyter_code>_, ax = plt.subplots(figsize=(8, 8))

for continent, color in ((0, 'red'), (1, 'blue')):
    df = insects_df[insects_df.continent == continent]
    ax.scatter(df.latitude, df.wingsize, s=40, c=color, label='Continent {}'.format(continent))

ax.set_xlabel('Latitude')
ax.set_ylabel('Wing Size')
ax.set_title('Are the two clusters associated with continent?')

ax.legend()<jupyter_output><empty_output><jupyter_text>Values of different continents seem scattered randomly across the two clusters, so it does **not** seem like continent is associated with the clusters.<jupyter_code>_, ax = plt.subplots(figsize=(8, 8))

for sex, color in ((0, 'red'), (1, 'blue')):
    df = insects_df[insects_df.sex == sex]
    ax.scatter(df.latitude, df.wingsize, s=40, c=color, label='sex {}'.format(sex))

ax.set_xlabel('Latitude')
ax.set_ylabel('Wing Size')
ax.set_title('Are the two clusters associated with sex?')

ax.legend()<jupyter_output><empty_output><jupyter_text>There we go!This is pretty definitive, the cluster of the larger insects are all female, and the cluster of smaller insects are all male.  This seems like enough evidence to conclude that the sex of the insect causes the data to cluster into two groups. Is an increasing latitude associated with a larger wing size?This question is a little more sophisticated, and we need some new technology to answer it.The idea is to create an equation:$$wingsize \approx a + b \cdot latitude$$Then we can look at the number $b$, which tells us how we should expect `wingsize` to change as `latitude` changes.  If we find that $b > 0$, that's evidence that an increasing latitude is associated with an increasing wingspan.I'll skip the technicalities, but the basic tool for creating equations like this is called **linear regression**.<jupyter_code>model = smf.ols(formula='wingsize ~ latitude', data=insects_df).fit()

model.summary()<jupyter_output><empty_output><jupyter_text>The linear regression estimated the equation as:$$wingsize \approx 765+ 2.54 \times latitude$$So we can expect, on average, an increase of 2.54 wing size for every additional latitude.But is this really meaningful?  We need to make sure our model is representing the data honestly.One way we can visualize this is to look at the equation as the equation for a line. If we *know* the latitude that we find an insect, we can *predict* the wing span using the equation.  If we plot the latitudes verses the predictions, we get a line.<jupyter_code>_, ax = plt.subplots(figsize=(8, 8))

for sex, color in ((0, 'red'), (1, 'blue')):
    df = insects_df[insects_df.sex == sex]
    ax.scatter(df.latitude, df.wingsize, s=40, c=color, label='sex {}'.format(sex))

# Draw the linear regression predictions
latitudes = np.linspace(30, 60, num=250)
wingsizes_hat = model.params[0] + model.params[1] * latitudes
ax.plot(latitudes, wingsizes_hat, linewidth=2, c='black')

ax.set_xlim(30, 60)
ax.set_xlabel('Latitude')
ax.set_ylabel('Wing Size')
ax.set_title('Insect wing sizes at various latitudes')
ax.legend()<jupyter_output><empty_output><jupyter_text>This plot demonstrates two serious flaws in our model:- The model has no knowledge of the sex of the insect, so the fit line attemps to bisect the two clusters of data.- The model cannot account for the curvature in the data points.  The model attempts to fit a line to data that does not have a linear shape. Accounting for the insects' sexOne way to account for the insects' sex is to modify our equation by adding a new term:$$wingsize \approx a + b \cdot latitude + c \cdot sex$$Again, linear regression can find an equation of this shape describing the data:<jupyter_code>model = smf.ols(formula='wingsize ~ latitude + sex', data=insects_df).fit()

model.summary()<jupyter_output><empty_output><jupyter_text>We now have an estimate for the number $c$ of $-88$.  This means, that on average, an insect with `sex = 1` costs it about $-88$ in wingsize.The predictions from this model now depend on whether an insect is male or female.  We have two lines of predictions, and the sex of the insect chooses which line to use:<jupyter_code>_, ax = plt.subplots(figsize=(8, 8))

for sex, color in ((0, 'red'), (1, 'blue')):
    df = insects_df[insects_df.sex == sex]
    ax.scatter(df.latitude, df.wingsize, s=40, c=color, label='sex {}'.format(sex))

# Draw the linear regression predictions
latitudes = np.linspace(30, 60, num=250)
for sex, color in ((0, 'red'), (1, 'blue')):
    wingsizes_hat = model.params[0] + model.params[1] * latitudes + model.params[2] * sex
    ax.plot(latitudes, wingsizes_hat, linewidth=2, c=color)

ax.set_xlim(30, 60)
ax.set_xlabel('Latitude')
ax.set_ylabel('Wing Size')
ax.set_title('Insect wing sizes at various latitudes')
ax.legend()<jupyter_output><empty_output><jupyter_text>Accounting for the curvature of the data pointsWe can account for the curvature of the data points by using a *poynomial regression*.  This means that we fit powers of latitude bigger than one:$$wingsize \approx a + b \cdot latitude + c \cdot latitude^2 +  d \cdot sex$$<jupyter_code>model = smf.ols(formula='wingsize ~ latitude + I(latitude**2) + sex', data=insects_df).fit()

model.summary()
_, ax = plt.subplots(figsize=(8, 8))

for sex, color in ((0, 'red'), (1, 'blue')):
    df = insects_df[insects_df.sex == sex]
    ax.scatter(df.latitude, df.wingsize, s=40, c=color, label='sex {}'.format(sex))

# Draw the linear regression predictions
latitudes = np.linspace(30, 60, num=250)
for sex, color in ((0, 'red'), (1, 'blue')):
    wingsizes_hat = model.params[0] + model.params[1] * latitudes + model.params[2] * latitudes ** 2 + model.params[3] * sex
    ax.plot(latitudes, wingsizes_hat, linewidth=2, c=color)

ax.set_xlim(30, 60)
ax.set_xlabel('Latitude')
ax.set_ylabel('Wing Size')
ax.set_title('Insect wing sizes at various latitudes')
ax.legend()<jupyter_output><empty_output><jupyter_text>This now looks like a good model that represents the data honestly. How is the line fitted?How did linear regression draw these lines and curves?  How were the coefficients estimated?We can think of the fitted line as a *prediction*.  If we were to collect a new insect at a certain latitude, the y-coordinate of the line would be our best estiamte for the wing size of that insect.A good strategy for drawing the line would seem to be: **draw the line that minimizes the dissimilarity between the predictions and the actual wing size.**<jupyter_code>model = smf.ols(formula='wingsize ~ latitude', data=insects_df).fit()

_, ax = plt.subplots(figsize=(8, 8))

for sex, color in ((0, 'red'), (1, 'blue')):
    df = insects_df[insects_df.sex == sex]
    ax.scatter(df.latitude, df.wingsize, s=40, c=color, label='sex {}'.format(sex))

latitudes = np.linspace(30, 60, num=250)
wingsizes_hat = model.params[0] + model.params[1] * latitudes
ax.plot(latitudes, wingsizes_hat, linewidth=2, c='black')

wingsizes_hat = model.params[0] + model.params[1] * insects_df.latitude
for latitude, wingsize, wingsize_hat in zip(insects_df.latitude, insects_df.wingsize, wingsizes_hat):
    ax.plot((latitude, latitude), (wingsize, wingsize_hat), color='grey')

ax.set_xlim(30, 60)
ax.set_xlabel('Latitude')
ax.set_ylabel('Wing Size')
ax.set_title('Insect wing sizes at various latitudes')
ax.legend()<jupyter_output><empty_output><jupyter_text>In the picture above the vertical distance between each data point $y$ and its estimated value $\hat{y}$ are highlighted.A common numeric measure of the dissimilarity is the *sum of squared residuals*:$$SSR = \sum_{i=1}^{N}{( y_i - \hat{y}_i )^2}$$**The linear regression line is the line that minimizes that sum of squared residuals.** Logistic RegressionWith linear regression we predict some quantity that has a continuous range of values.  *Logistic Regression* solves a slightly different problem.  What if we wanted to predict the sex of an insect from the other measurements?<jupyter_code>_, ax = plt.subplots(figsize=(8, 4))

# Scatterplot of the data
ax.scatter(insects_df.latitude, insects_df.sex)
ax.set_xlim([30, 65])
ax.set_yticks([0, 1])
ax.set_yticklabels(['sex = 0', 'sex = 1'])
ax.set_xlabel('Latitude')<jupyter_output><empty_output><jupyter_text>Logistic regression attempts to estimate the probability that an insect is male ($sex = 1$) or female ($sex = 0$).$$P(sex = 1 \mid latitude)$$<jupyter_code>model = smf.logit(formula='sex ~ latitude', data=insects_df).fit()

model.summary()<jupyter_output>Optimization terminated successfully.
         Current function value: 0.603191
         Iterations 5<jupyter_text>Logistic regression attempts to draw a *curve* of predicted probabilities (not a line, as in linear regression).The above logistic regression results in the following curve:$$P(sex = 1 \mid latitude) = \frac{1}{1 + e^{-(7.3 - 0.16 \times latitude)}}$$<jupyter_code>_, ax = plt.subplots(figsize=(8, 4))

ax.scatter(insects_df.latitude, insects_df.sex)

df = pd.DataFrame({'latitude': np.linspace(30, 65, num=250)})
ax.plot(df, model.predict(df), linewidth=2)

ax.set_xlim([30, 65])
ax.set_yticks([0, 1])
ax.set_yticklabels(['sex = 0', 'sex = 1'])
ax.set_xlabel('Latitude')
ax.set_ylabel('$P(sex = 1)$')<jupyter_output><empty_output><jupyter_text>What if we need to make a binary decision, i.e., we need to use the estimated probabilities to actually classify each insect as either male or female?We can accomplish this by thresholding probabilities:<jupyter_code>fig, ax = plt.subplots(figsize=(8, 4))

# Scatterplot of the data
ax.scatter(insects_df.latitude, insects_df.sex)

df = pd.DataFrame({'latitude': np.linspace(30, 65, num=250)})
ax.plot((30, 65), (0.5, 0.5), color='black', linestyle='--')
ax.plot((47.075, 47.075), (0, 1), color='black', linestyle='--')
ax.plot(df, model.predict(df), linewidth=2)

ax.set_xlim([30, 65])
ax.set_yticks([0, 1])
ax.set_yticklabels(['sex = 0', 'sex = 1'])
ax.set_xlabel('Latitude')
ax.set_ylabel('$P(sex = 1)$')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11069805865344899, 'fraction_numerical': 0.02664188351920694, 'mean_word_length': 5.354768153980753, 'pattern_counts': {'":': 0, '<': 75, '<?xml version=': 0, '>': 76, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 5, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '214665', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 4935, 'n_tokens_neox': 4562, 'n_words': 1707, 'path': 'day-2/basics-of-machine-learning.ipynb', 'repo_name': 'GalvanizeOpenSource/statistics-workshop', 'size': '14526'}