<jupyter_start><jupyter_text>====================================================== MA477 - Theory and Applications of Data Science   Project 1: Linear Regression & KNN Regressor   Dr. <NAME>  United States Military Academy, West Point, AY20-2======================================================= Weight: 75pts     Due Date: Feb. 19th by COB  Cadet Name:Date:  $\dots \dots$ MY DOCUMENTATION IDENTIFIES ALL SOURCES USED AND ASSISTANCE RECEIVED IN THIS ASSIGNMENT$\dots \dots$  I DID NOT USE ANY SOURCES OR ASSISTANCE REQUIRING DOCUMENATION IN COMPLETING THIS ASSIGNMENT Signature/Initials:  Description of ProjectIn this project you will be working with the `incurance` dataset contained in the folder. The `insurance` dataset contains information of different people, and based on this information how much the insurance compay charges them. The ultimate goal is to predict the insurance charges for new people based on the historical information collected about each person. The features that have been collected are: `age, sex, bmi, children, smoker,` and `region`. You are provided with two separate sets: `insurance_train` which you will use to train your model and `insurance_test` which you will use to generate the predictions. Once you generate the predictions, you should save the results on an excel file and submit that to me. The project will be broken into two main parts: The Analysis and  Best Model portions.AnalysisThe Analysis portion is worth 45pts.For this portion of the project you will be evaluated on the overall analysis of the dataset.The rough point breakdown is as follows:A one or two pargraph explaining what the project is about, what are the results, and the methodology: 10ptsThoroughly addressing all the questions and completing all the required tasks: 20pts  Creativity and quality of Python code and explanation of the step-by-step code/work: 15ptsTasks     Data Preprocessing: Explain how you are handling the missing data, are you scaling the data and why?        Conduct exploratory analysis, and briefly summarize your observations and findings. Exploring your data using a variety of visual tools counts as part of exploratory analysis.            Explain how you handled each of the qualitative features.        What model did you pick and why? At this stage only choose between KNN Regressor and Linear Regression      Explain the type of relationship(linear vs. non-linear) between features and response variable, and how you came to your conclusions.        Obtain a measure of your model's variability and prediction power. Explain your method and results.        Best ModelThe Best Model portion is worth 30pts.For this portion you will exclusively be evaluated on the predictive power of your model. In this portion you will be competing with the rest of your peers for the top score. The scores will be, roughly, distributed according to the following scheme:There are two instances where there may be significant deviations from the above grading scheme:      Points    Criteria           28-30    Highest R2 Score           26-28    .25 StDevs of highest R2 score               24-26    .5 StDevs of highest R2 score          22-24    .75 StDevs of highest R2 score          19-22    1.25 StDevs of highest R2 score          17    1.75 StDevs of highest R2 score          15    2.25 StDevs of highest R2 score           7-10     2.25+ StDevs of highest R2 score       If a student's model achieves an R2 score that is at least as high as the R2 score achieved by the model build by the instructor, then you will automatically get a $30$   If the highest R2 score by a student is signigicantly smaller than the R2 score achieved by the model built by the instructor, the insturctor reserves the right to assign a max score that is significantly lower than what is stated in the grading scheme above. In this situation the rest of the scores will also be adjusted accordingly.   How to Submit?  All of your work should be done in a single JupyterNotebook. You will submit to me via email a total of two files. One will be the JupyterNotebook which contains your whole work. The second file should be an excel file containing ONLY the predictions of your model on the test set `insurance_test`. BackgroundWith Machine Learning, we can use data to make predictions. Two of these methods are K Nearest Neighborhood Regression and Linear Regression. In this project, I will use these two methods to predict the cost of insurance based on a person's age, gender, BMI index, number of children, whether they're a smoker, and what region they live in. To determine which method to use and which variables to use, we need to do some exploratory analysis on the data.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns<jupyter_output><empty_output><jupyter_text>First, I downloaded the dataset and examined what kind of data we have<jupyter_code>ins_t = pd.read_csv('insurance_train.csv')
ins_test = pd.read_csv('insurance_test.csv')
ins_t.head()
ins_t['region'].unique()<jupyter_output><empty_output><jupyter_text>Exploratory AnalysisIt looks like there is some missing data in a few of the columns and there are three qualitative measures, so we will have to deal with these by converting the sex to 1 for males and 0 for females. Smokers will be 1 and nonsmokers will be 0. Finally, people in the northwest region will be 0, northest 1, southwest 2, and southeast will be 3.<jupyter_code>def sex(x):
    if x == 'male':
        return(1)
    else:
        return(0)
    
def smoker(x):
    if x == 'yes':
        return(1)
    else:
        return(0)
    
def region(x):
    if x == 'northwest':
        return(0)
    elif x == 'northeast':
        return(1)
    elif x == 'southwest':
        return(2)
    else:
        return(3)
df = ins_t.copy()
df['sex'] = ins_t['sex'].apply(sex)
df['smoker'] = ins_t['smoker'].apply(smoker)
df['region'] = ins_t['region'].apply(region)
dfn = df.copy()
df.head()<jupyter_output><empty_output><jupyter_text>To deal with the missing data, first I am going to explore some of the qualities of the missing data. Like how much data is missing, and what range the data could be in.<jupyter_code>df.shape<jupyter_output><empty_output><jupyter_text>To find out if there is a missing data point in the column, I used this function to count them. I used the website https://stackoverflow.com/questions/944700/how-can-i-check-for-nan-values to find out whether a data point is missing or not using the 'isnan' method.<jupyter_code>def cntna(x):
    tot = 0
    for n in x:
        if np.isnan(n) == True:
            tot += 1
    return(tot)
cntna(df['bmi'])
df['bmi'].min()
df['bmi'].max()
df['bmi'].mean()
df['bmi'].std()
cntna(df['age'])
df['age'].min()
df['age'].max()
df['age'].mean()
df['age'].std()<jupyter_output><empty_output><jupyter_text>First, I'm going to just replace the missing data from the BMI and age columns with their averages.<jupyter_code>def replacebmi(x):
    if np.isnan(x) == True:
        return(30.630)
    else:
        return(x)
    
def replaceage(x):
    if np.isnan(x) == True:
        return(39.0)
    else:
        return(x)
df['bmi'] = ins_t['bmi'].apply(replacebmi)
df['age'] = ins_t['age'].apply(replaceage)
df.head()<jupyter_output><empty_output><jupyter_text>Looking at the plots after these changes, I realize that making all the missing data the average of their respective columns really skew the data, because there are a significant amout of data points at the average values.<jupyter_code>sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='age',y='charges',data=df,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()
sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='bmi',y='charges',data=df,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()<jupyter_output><empty_output><jupyter_text>To try something new, I came up with the idea to assign a random value for the missing data based on the distribution of the know values in the same columns. The new data depends on the average, range, and standard deviation of the known data. I used the website https://stackoverflow.com/questions/27831923/python-random-number-generator-with-mean-and-standard-deviation to come up with an array of random numbers within a certain distribution.<jupyter_code>import scipy.stats as stats

a, b = 18, 64
mu, sigma = 39.1025, 13.3337
agedist = stats.truncnorm((a - mu) / sigma, (b - mu) / sigma, loc=mu, scale=sigma)

agevalues = agedist.rvs(86)

c, d = 15.96, 53.13
bmu, bsigma = 30.6304, 6.0226
bmidist = stats.truncnorm((c - bmu) / bsigma, (d - bmu) / bsigma, loc=bmu, scale=bsigma)

bmivalues = bmidist.rvs(49)<jupyter_output><empty_output><jupyter_text>I used these functions to change the replace the missing data with a new value from the distribution.<jupyter_code>import random as ran
def replacebmi2(x):
    if np.isnan(x) == True:
        return(bmivalues[ran.randint(0, 48)])
    else:
        return(x)

def replaceage2(x):
    if np.isnan(x) == True:
        return(agevalues[ran.randint(0, 85)])
    else:
        return(x)
dfn['bmi'] = ins_t['bmi'].apply(replacebmi2)
dfn['age'] = ins_t['age'].apply(replaceage2)
dfn.head()<jupyter_output><empty_output><jupyter_text>After I made these changes, the plots looked more natural and I could begin looking to see if there is a relationship between the variables and the insurance charge. The two most obvious linear relationships are with age and charges and with smokers and charges, so these two variables are the ones I used in creating my first model. There doesn't seem to be any other relationships other than linear relationships because there doesn't look like the data follows a quadratic equation.<jupyter_code>sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='age',y='charges',data=dfn,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()
sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='bmi',y='charges',data=dfn,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()
sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='children',y='charges',data=dfn,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()
sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='smoker',y='charges',data=dfn,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()
sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='region',y='charges',data=dfn,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()
sns.set_style('whitegrid')
plt.figure(figsize=(10,8))
sns.regplot(x='sex',y='charges',data=dfn,scatter_kws={'s':6},line_kws={'color':'r'},ci=95)
plt.show()<jupyter_output><empty_output><jupyter_text>I created a new dataframe with only the smoker and age variables.<jupyter_code>dfn.head()
dff = dfn.drop(['sex','bmi','children','region'],axis=1)
dff.head()
from sklearn.preprocessing import StandardScaler
X=dff.drop('charges',axis=1)
y=dff['charges']
X.head()<jupyter_output><empty_output><jupyter_text>Before I begin creating each of the models, I am scaling the data because each of the variables are measured on different scales and it is important to elimainate the false effect that having larger or smaller distances between data points has when using machine learning to create a model.<jupyter_code>scaler=StandardScaler()
scaled=scaler.fit_transform(X)
X_sc=pd.DataFrame(scaled,columns=X.columns,index=X.index)
X_sc.head()
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test=train_test_split(X_sc,y,test_size=0.25, random_state=101)
X_train.head()
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import r2_score, mean_squared_error<jupyter_output><empty_output><jupyter_text>The first method I used was KNN Regression. Here, I tuned the model to see which K value would yield the highest R2 score. The R2 score gives us a measure for the prediction power of our model. The closer it is to 1, the better. The variablilty of the model can be seen by looking at the consistency of the R2 scores. Since they aren't all over the place, we can conclude that the model has good variability.<jupyter_code>knn_r2scores=[]

for i in range(1,100):
    knn=KNeighborsRegressor(n_neighbors=i)
    knn.fit(X_train,y_train)
    pred=knn.predict(X_test)
    r2=r2_score(y_test,pred)
    knn_r2scores.append(r2)
sns.set_style('darkgrid')
plt.figure(figsize=(8,6))
plt.plot(knn_r2scores,c='r')
plt.xlabel("K value",fontsize=14)
plt.ylabel("R2 Score",fontsize=14)<jupyter_output><empty_output><jupyter_text>It looks like a K value of 20 would give us the best R2 score. Next, I used linear regression to come up with a new model.<jupyter_code>from sklearn.linear_model import LinearRegression
lg=LinearRegression()
lg.fit(X_train,y_train)
lg.intercept_
lg.coef_
lg_pred=lg.predict(X_test)
sns.scatterplot(y_test,lg_pred)
lg_r2=r2_score(y_test,lg_pred)
mse=mean_squared_error(y_test,lg_pred)<jupyter_output><empty_output><jupyter_text>The R2 score that I got using this method was worse than the R2 score usign KNN.<jupyter_code>lg_r2
mse<jupyter_output><empty_output><jupyter_text>I wanted to see if the data was truly linear, so I plotted the distribution of the error from the model. Under the assumptions of linear regression, the error is normally distributed. If the model is not linear, then the error will not be normally distributed. From the plot below, the error looks pretty normally distributed other than the humps on either side of the curve.<jupyter_code>plt.figure(figsize=(8,6))
sns.distplot((y_test-lg_pred), bins=25)<jupyter_output><empty_output><jupyter_text>I wanted to see if I would get a better R2 score using all the variables that were collected. The results show that we would indeed get a better R2 score for both KNN and linear regression.<jupyter_code>dff2 = dfn.copy()
X2=dff2.drop('charges',axis=1)
y2=dff2['charges']
scaled2=scaler.fit_transform(X2)
X2_sc=pd.DataFrame(scaled2,columns=X2.columns,index=X2.index)
X2_sc.head()
X2_train,X2_test,y2_train,y2_test=train_test_split(X2_sc,y2,test_size=0.25, random_state=101)
knn2_r2scores=[]

for i in range(1,100):
    knn=KNeighborsRegressor(n_neighbors=i)
    knn.fit(X2_train,y2_train)
    pred=knn.predict(X2_test)
    r2=r2_score(y2_test,pred)
    knn2_r2scores.append(r2)
sns.set_style('darkgrid')
plt.figure(figsize=(8,6))
plt.plot(knn2_r2scores,c='r')
plt.xlabel("K value",fontsize=14)
plt.ylabel("R2 Score",fontsize=14)<jupyter_output><empty_output><jupyter_text>The best R2 score I got with KNN was around 0.79 and the best I got with linear regression was 0.72. It seems like KNN would be the best method to use because it has consistently had a better R2 score than linear regression.<jupyter_code>lg.fit(X2_train,y2_train)
lg.coef_
lg2_pred=lg.predict(X2_test)
sns.scatterplot(y2_test,lg2_pred)
lg2_r2=r2_score(y2_test,lg2_pred)
mse=mean_squared_error(y2_test,lg2_pred)
lg2_r2
mse
plt.figure(figsize=(8,6))
sns.distplot((y2_test-lg2_pred), bins=25)<jupyter_output><empty_output><jupyter_text>Another method that we could use is a shrinkage method called ridge regression where we constrain the estimated coefficients.<jupyter_code>from sklearn.model_selection import cross_validate, KFold
from sklearn.linear_model import Ridge, Lasso, LinearRegression
X2_sc.head()
ridge=Ridge(alpha=1.25)
kf=KFold(n_splits=5,shuffle=True, random_state=101)
cv_ridge=cross_validate(estimator=ridge,X=X2_sc,y=y2,cv=kf, return_estimator=True,
                        scoring={'r2','neg_mean_squared_error'})
import sklearn
cv_ridge['test_r2']
cv_ridge['test_r2'].mean()<jupyter_output><empty_output><jupyter_text>When we cross validate the ridge method, the average R2 score was 0.74, which is worse than what we got with KNN. The final regression method that I wanted to use was KNN with fewer of the variables, so I fit the KNN model using only age, BMI, and smoker. The K value of 9 gave us the highest R2 score we had, so I decided to go with this to predict the charges from the insurance test dataset.<jupyter_code>dff3 = dfn.copy()
X3=dff3.drop(['charges','sex','children','region'],axis=1)
y3=dff3['charges']
scaled3=scaler.fit_transform(X3)
X3_sc=pd.DataFrame(scaled3,columns=X3.columns,index=X3.index)
X3_sc.head()
X3_train,X3_test,y3_train,y3_test=train_test_split(X3_sc,y3,test_size=0.25, random_state=101)
knn3_r2scores=[]

for i in range(1,100):
    knn=KNeighborsRegressor(n_neighbors=i)
    knn.fit(X3_train,y3_train)
    pred=knn.predict(X3_test)
    r2=r2_score(y3_test,pred)
    knn3_r2scores.append(r2)
sns.set_style('darkgrid')
plt.figure(figsize=(8,6))
plt.plot(knn3_r2scores,c='r')
plt.xlabel("K value",fontsize=14)
plt.ylabel("R2 Score",fontsize=14)
knn=KNeighborsRegressor(n_neighbors=9)
knn.fit(X3_train,y3_train)
pred=knn.predict(X3_test)
r2=r2_score(y3_test,pred)
mse=mean_squared_error(y3_test,pred)
r2
mse<jupyter_output><empty_output><jupyter_text>Interpreting the test MSE value is harder, but essentially, we want it to be as small as possible. Now, I used the last KNN regressor to predict the values of the charges for the insurance test dataset and then exported them in an excel document.<jupyter_code>ins_test.head()
df_test = ins_test.copy()
df_test = df_test.drop(['sex','children','region'],axis=1)
df_test['smoker'] = ins_test['smoker'].apply(smoker)
df_test.head()
cntna(df_test['bmi'])
cntna(df_test['age'])
scaled_test=scaler.fit_transform(df_test)
X_sc_test = pd.DataFrame(scaled_test,columns=df_test.columns,index=df_test.index)
X_sc_test.head()
test_pred=knn.predict(X_sc_test)
test_predictions=pd.DataFrame(test_pred)
test_predictions.to_excel('Project 1 Predictions.xlsx')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1051861028211048, 'fraction_numerical': 0.02348105186102821, 'mean_word_length': 5.383896477354421, 'pattern_counts': {'":': 0, '<': 86, '<?xml version=': 0, '>': 86, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '158604', 'license': 'MIT', 'n_tokens_mistral': 6093, 'n_tokens_neox': 5786, 'n_words': 1910, 'path': 'MA477 - Theory and Applications of Data Science/Homework/Student Solutions/Project 1/Project 1 - Hudson.ipynb', 'repo_name': 'jkstarling/MA477-copy', 'size': '17765'}