<jupyter_start><jupyter_text>Lab 3_01: Statistical Modeling and Model Validation> Authors: <NAME>, <NAME>---  ObjectiveThe goal of this lab is to guide you through the modeling workflow to produce the best model you can. In this lesson, you will follow all best practices when slicing your data and validating your model.   Imports<jupyter_code># Import everything you need here.
# You may want to return to this cell to import more things later in the lab.
# DO NOT COPY AND PASTE FROM OUR CLASS SLIDES!
# Muscle memory is important!

import pandas as pd
from scipy.stats import ttest_ind
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score, mean_squared_error

import matplotlib.pyplot as plt

import statsmodels.api as sm

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Read DataThe `citibike` dataset consists of Citi Bike ridership data for over 224,000 rides in February 2014.<jupyter_code># Read in the citibike data in the data folder in this repository.
citibike = pd.read_csv('../data/citibike_feb2014.csv')<jupyter_output><empty_output><jupyter_text>Explore the dataUse this space to familiarize yourself with the data.Convince yourself there are no issues with the data. If you find any issues, clean them here.<jupyter_code># Check first five rows.
citibike.head()
# Check datatypes and numbers of non-null values.
citibike.info()
# Summarize all variables.
citibike.describe(include='all')
# Check for missing values. (This is easier to read 
# than the .info() output.)
citibike.isnull().sum()<jupyter_output><empty_output><jupyter_text>Is average trip duration different by gender?Conduct a hypothesis test that checks whether or not the average trip duration is different for `gender=1` and `gender=2`. Be sure to specify your null and alternative hypotheses, and to state your conclusion carefully and correctly! $$\begin{eqnarray*}&H_0:& \mu_1 = \mu_2 \\&H_A:& \mu_1 \neq \mu_2\end{eqnarray*}$$We will conduct this test assuming $\alpha=0.05$.<jupyter_code>ttest_ind(citibike[citibike['gender'] == 1]['tripduration'],
          citibike[citibike['gender'] == 2]['tripduration'])<jupyter_output><empty_output><jupyter_text>**Answer**: Our $t$-statistic is -5.93 and our $p$-value is very, very small. Because $p$ is smaller than $\alpha$, we reject our null hypothesis and accept that $\mu_1 \neq \mu_2$. This means that we accept that the average trip duration is different for `gender=1` and `gender=2`.  What numeric columns shouldn't be treated as numeric? **Answer:** The `start station id`, `end station id`, and `bikeid` columns are all categorical in nature (e.g. adding two of these ID numbers together would be meaningless). These are technically integers, but should not be treated that way.  Dummify the `start station id` Variable<jupyter_code># Before dummifying, let's see how many columns we should create.
len(set(citibike['start station id']))
# How many columns are there in the original data?
len(citibike.columns)
# Let's dummy the data.
citibike = pd.get_dummies(citibike,
                          columns=['start station id'],
                          drop_first=True)
# How many columns are there now?
len(citibike.columns)
# 329 unique values + 15 original columns  = 344.
# We dropped the `start station id` variable: 344 - 1 = 343.
# We set `drop_first = True`: 343 - 1 = 342.

# We got the right number of columns in our output!

# Let's check out our data to make sure it looks like we did this right.
citibike.head()<jupyter_output><empty_output><jupyter_text>Engineer a feature called `age` that shares how old the person would have been in 2014 (at the time the data was collected).- Note: you will need to clean the data a bit.<jupyter_code># First attempt.
citibike['age'] = 2014 - citibike['birth year']
# We got an error! Somewhere, there's a string.

# Check the values in the birth year column.
citibike['birth year'].value_counts()
# Can we just pull out the strings?

# Iterate through all unique values in birth year column.
for i in set(citibike['birth year']):

    # Try typecasting each value to be an integer.
    try:
        int(i)
        
    # If it gives you an error (so it can't be
    # printed as an integer), print the value.
    except:
        print(i)
# How many values of "\N" are there?
citibike[citibike['birth year'] == '\N'].shape[0]
# How many values of "\N" are there?
# We got an error - it interprets \ as an escape character.
# We need to use the escape character twice!
citibike[citibike['birth year'] == '\\N'].shape[0]
# There's 6,717 values, which is just under 3% of the rows.
# Let's replace "\N" with np.nan.

citibike.loc[citibike['birth year'] == '\\N','birth year'] = np.nan
# Did we successfully do this?
citibike.isnull().sum()
# Now let's try creating our age column.
citibike['age'] = citibike['birth year'].map(lambda x: 2014 - int(x),
                                             na_action = 'ignore')
# Let's check to see if age and birth year seem to match up.
citibike['age'].hist();
citibike['birth year'].dropna().astype(int).hist();
# Yes, birth year is a mirror image of age.<jupyter_output><empty_output><jupyter_text>Split your data into train/test dataLook at the size of your data. What is a good proportion for your split? **Justify your answer.**Use the `tripduration` column as your `y` variable.For your `X` variables, use `age`, `usertype`, `gender`, and the dummy variables you created from `start station id`. (Hint: You may find the Pandas `.drop()` method helpful here.)**NOTE:** When doing your train/test split, please use random seed 123.<jupyter_code># Because usertype is a column of strings, we must
# dummy that column as well.
citibike = pd.get_dummies(citibike,
                          columns=['usertype'],
                          drop_first=True)
X_train, X_test, y_train, y_test = train_test_split(citibike.dropna().drop(columns=['tripduration', 'birth year', 'bikeid',
                                                                           'end station longitude', 'end station latitude',
                                                                           'end station name', 'end station id',
                                                                           'start station longitude', 'start station latitude',
                                                                           'start station name', 'starttime', 'stoptime']),
                                                    citibike.dropna()['tripduration'],
                                                    test_size=0.2,
                                                    random_state=123)
X_train.head()
X_test.shape<jupyter_output><empty_output><jupyter_text>**Answer**: The more data we train on, the better it will usually perform! I used `test_size = 0.2` because we have lots of data. This leaves a lot of data (about 43,600 rows!) in our test set to still evaluate our model.  Fit a Linear Regression model in `sklearn` predicting `tripduration`.<jupyter_code># Step 1. Instantiate the model.
model = LinearRegression()

# Step 2. Fit the model on the training data.
model.fit(X_train, y_train)

# Step 3. Generate predictions.
preds = model.predict(X_test)<jupyter_output>/Users/mattbrems/anaconda3/lib/python3.6/site-packages/sklearn/linear_model/base.py:485: RuntimeWarning: internal gelsd driver lwork query error, required iwork dimension not returned. This is likely the result of LAPACK bug 0038, fixed in LAPACK 3.2.2 (released July 21, 2010). Falling back to 'gelss' driver.
  linalg.lstsq(X, y)<jupyter_text>Evaluate your modelLook at some evaluation metrics for **both** the training and test data. - How did your model do? Is it overfit, underfit, or neither?- Does this model outperform the baseline? (e.g. setting $\hat{y}$ to be the mean of our training `y` values.)<jupyter_code># Check the MSE on the training and testing sets.

print(f'MSE on testing set: {mean_squared_error(y_train, model.predict(X_train))}')
print(f'MSE on training set: {mean_squared_error(y_test, preds)}')
# Check the R^2 on the training and testing sets.

print(f'R^2 on testing set: {r2_score(y_train, model.predict(X_train))}')
print(f'R^2 on training set: {r2_score(y_test, preds)}')<jupyter_output>R^2 on testing set: 0.003732147945243902
R^2 on training set: -0.004823021824723606<jupyter_text>**Answer**:  Based on the MSE, our model is performing far worse on the testing set than on the training set, which means that our model is likely overfit to the data.Based on the $R^2$, our model is explaining approximately zero variance in the $Y$ data. Our model is probably quite bad.<jupyter_code>plt.figure(figsize = (12, 9))

# Examine the relationship between observed and predicted values.
plt.scatter(y_test, preds)

# Line showing perfect predictions.
plt.plot([0, max(max(y_test),max(preds))],
         [0, max(max(y_test),max(preds))],
         linestyle = '--')

plt.title('Predicted values are quite small,\nbut true values are spread out!', fontsize = 24)
plt.xlabel('True Values', fontsize = 16)
plt.ylabel('Predicted Values', fontsize = 16);
print(f'MSE of baseline model: {mean_squared_error(y_test, [np.mean(y_train)] * len(y_test))}')
print(f'R^2 of baseline model: {r2_score(y_test, [np.mean(y_train)] * len(y_test))}')<jupyter_output>R^2 of baseline model: -0.00024026139601152074<jupyter_text>**Answer**:  Based on the above information, I conclude that my model is both overfit to the data and a bad model.- Our MSE and $R^2$ comparing our observed `y_test` values to the average `y_train` value are better than the MSE and $R^2$ on the more complex model we've fit on the training dataset!- I might try removing features to improve the fit of the model.  Fit a Linear Regression model in `statsmodels` predicting `tripduration`.<jupyter_code># Remember, we need to add a constant in statsmodels!
X_train = sm.add_constant(X_train)
model_sm = sm.OLS(y_train, X_train).fit()<jupyter_output><empty_output><jupyter_text>Using the `statsmodels` summary, test whether or not `age` has a significant effect when predicting `tripduration`.- Be sure to specify your null and alternative hypotheses, and to state your conclusion carefully and correctly **in the context of your model**!<jupyter_code>model_sm.summary()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09933774834437085, 'fraction_numerical': 0.019003743161531816, 'mean_word_length': 3.72562358276644, 'pattern_counts': {'":': 0, '<': 48, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '489861', 'license': 'Apache-2.0', 'n_tokens_mistral': 3247, 'n_tokens_neox': 3025, 'n_words': 1214, 'path': 'Notebook/Lab-regression-and-model-validation/solution-code/solution-code.ipynb', 'repo_name': 'sgdataguru/ITE_Machine_Learning_Workshop', 'size': '10425'}