<jupyter_start><jupyter_text>Module 10 Submission  Machine Learning and Data Mining <NAME>  7/20/2020  Polynomial Feature Selection In this module, you will write one Python notebooks to create required outputs.             Watch the lecture videos and fully understand the lecture notebook    Analyze the “poly_data” data set by following the example in the lecture    Add code in the assignment template and generate required outputs    Submission: You will export your notebook to both .html and .py formats. You will submit the following file to Blackboard. In your html file, you should include only required outputs of your python script without error messages.     Firstname_Lastname_polymodel_selection.zip (zip the .html and .ipynb files) Attachments:     Model_evaluation_lecture.ipynb: the notebook from the lecture    Polynomial_feature_selection_assignment_template.ipynb: the assignment template    Poly_data.csvPreview the document: the data set    Assignment 10.pdfPreview the document<jupyter_code># Import packages
import time
import numpy as np
from matplotlib import pyplot as plt
import seaborn as sns
from pandas import Series, DataFrame
import pandas as pd
%matplotlib inline
#Read in data from a data file to data_df in DateFrame format
csv = pd.read_csv("poly_data.csv")
data_df= pd.DataFrame(csv)

#verify the dataframe is imported correctly 
print(data_df.head(6))
#joint plot (or scatter plot) of X1 and y
sns.jointplot(data_df['X1'], data_df['y'])
#joint plot (or scatter plot) of X2 and y
sns.jointplot(data_df['X2'], data_df['y'])
#joint plot (or scatter plot) of X1 and X2
sns.jointplot(data_df['X1'], data_df['X2'])<jupyter_output><empty_output><jupyter_text>Based on observing the above 3 diagrams and the p-values displayed, we found both X1 and X2 have close correlation with y. X1 and X2 are independent from each other.<jupyter_code># split the data into training and testing datasets
# the percentage of training data is 75%

#split point 
percentage_for_training = 0.75

# Count training data
number_of_training_data = int(len(data_df)*percentage_for_training)

#create training and testing datasets
train_df  = data_df[0:number_of_training_data]
test_df = data_df[number_of_training_data:]
print(train_df.shape)
print(test_df.shape)
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

#set the degree to 3, you can try a larger number if you like
#for degree = 3, we will generate 9 features. 
polynomial_features = PolynomialFeatures(degree=3)
X_poly = polynomial_features.fit_transform(data_df[['X1','X2']])

X_train = X_poly[0:number_of_training_data]
X_test = X_poly[number_of_training_data:]
# mse() calculates mean square error of a model on given X and y
def mse(X, y, model):
    return  ((y-model.predict(X))**2).sum()/y.shape[0]
# use all the features to train the linear model 
lm = LinearRegression()
lm.fit(X_train, train_df['y'])
train_mse = mse(X_train, train_df['y'], lm)
print("Training Data Set's MSE is: \t", train_mse)
test_mse = mse(X_test, test_df['y'], lm)
print("Testing Data Set's MSE is : \t", test_mse)
#import lasso
#lasso is controlled by a parameter alpha.
#by fine tuning this parameter, we can control the number of features

from sklearn.linear_model import Lasso
#Train the model, try different alpha values.
Lasso_model = Lasso(alpha=0.15,normalize=True, max_iter=1e5, )
Lasso_model.fit(X_train, train_df['y'])
        
#see the trained parameters. Zero means the feature can be removed from the model
Lasso_model.coef_
#let's see the train_mse and test_mse from Lasso when 
#alpha = 0.15

train_mse = mse(X_train, train_df['y'], Lasso_model)
print("Training Data Set's MSE is: \t", train_mse)
test_mse = mse(X_test, test_df['y'], Lasso_model)
print("Testing Data Set's MSE is : \t", test_mse)
#let's try a large range of values for alpha first
#create 50 alphas from 100 to 0.00001 in logspace
alphas = np.logspace(2, -5, base=10, num=50)
alphas
#use arrays to keep track of the MSE of each alpha used. 
train_mse_array =[]
test_mse_array=[]

#try each alpha
for alpha in alphas:
    
    #create Lasso model using alpha
    Lasso_model = Lasso(alpha=alpha,normalize=True, max_iter=1e5, )
    Lasso_model.fit(X_train, train_df['y'])
    
    #Calculate MSEs of train and test datasets 
    train_mse = mse(X_train, train_df['y'], Lasso_model)
    test_mse = mse(X_test, test_df['y'], Lasso_model)
    
    #add the MSEs to the arrays
    train_mse_array.append(train_mse)
    test_mse_array.append(test_mse)
    
#plot the MSEs based on alpha values
#blue line is for training data
#red line is for the testing data
plt.plot(np.log10(alphas), train_mse_array)
plt.plot(np.log10(alphas), test_mse_array, color='r')<jupyter_output><empty_output><jupyter_text>There is something interesting between 0 and 1 in the above diagram. 0 mean 10^0=1 While 1 means 10^1 = 10  so, we will look closely within this range to find the optimal alpha value<jupyter_code># We can try a smaller search space now (a line space between 1 and 10)
alphas = np.linspace(1, 10, 1000)
train_mse_array =[]
test_mse_array=[]

for alpha in alphas:
    
    Lasso_model = Lasso(alpha=alpha, normalize=True, max_iter=1e5)
    Lasso_model.fit(X_train, train_df['y'])
                    
    train_mse = mse(X_train, train_df['y'], Lasso_model)
    test_mse = mse(X_test, test_df['y'], Lasso_model)
                    
    train_mse_array.append(train_mse)
    test_mse_array.append(test_mse)
    
# Find closest distance possible between train and test
train = np.array(train_mse_array)
test = np.array(test_mse_array)
value = abs(train-test)
minimum = np.min(value)

# Find the index where they are closest, this gives us alpha
index = np.where(value==minimum)

plt.plot(alphas, train_mse_array)
plt.plot(alphas, test_mse_array, color='r')

print("The optimal alpha is: ", alphas[index])
print("Train MSE is: ", train_mse_array[index])
print("Test MSE is: ", test_mse_array[index])<jupyter_output>The optimal alpha is:  [3.55855856]
Train MSE is:  48347.943216117455
Test MSE is:  34593.52285633148<jupyter_text>By observing a smaller range of alpha, we can clearly see how the MSEs change as we change the model and features. Use the diagram to explain the trends of the two lines and summarize what you learned so far.<jupyter_code>## type your code here to describe the above diagram and what you learned 
## so far about feature and model selection ( about 200 words )
############################################################################
print("First, to describe the diagram above we see that as alpha increases, our Train and Test MSE converge until alpha "
     + "reaches the its optimal value of ~3.5586. At this point we see our Train and Test MSE overlap as they begin to "
     + "separate. Our Train MSE continues upwards with an increasing slope, while the Test MSE reaches down to ~34,000 where it "
     + "has the slope return to positive. What we find is that the features we choose have a major impact on the traning of our model. "
     + "we don't need two features that are highly correlated, as this is redundant, adds no benefit to the model, increases noise, and "
     + "could lead to our model being training on irrelevant data. Simplying our model to reduce the features and in turn the redundancy "
     + "means our model has less of a chance of being taught to make decisions based on noise. Plus less features is quicker computations! "
     + "Furthermore, we learned efficient ways to remove features that are highly correlated and unnecessary in the model. Using heatmaps and "
     + "matrixes allow us to quickly and visually identify what is 'out of place' so it can be dropped from our data.")
#############################################################################<jupyter_output>First, to describe the diagram above we see that as alpha increases, our Train and Test MSE converge until alpha reaches the its optimal value of ~3.5586. At this point we see our Train and Test MSE overlap as they begin to separate. Our Train MSE continues upwards with an increasing slope, while the Test MSE reaches down to ~34,000 where it has the slope return to positive. What we find is that the features we choose have a major impact on the traning of our model. we don't need two features that are highly correlated, as this is redundant, adds no benefit to the model, increases noise, and could lead to our model being training on irrelevant data. Simplying our model to reduce the features and in turn the redundancy means our model has less of a chance of being taught to make decisions based on noise. Plus less features is quicker computations! Furthermore, we learned efficient ways to remove features that are highly correlated and unnecessary in the model. Using heatmaps and matrixe[...]
{'fraction_non_alphanumeric': 0.08925339366515837, 'fraction_numerical': 0.01821266968325792, 'mean_word_length': 4.759609120521173, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '408805', 'license': 'MIT', 'n_tokens_mistral': 2777, 'n_tokens_neox': 2621, 'n_words': 1118, 'path': '7 Polynomial Regression and Model Evaluation/Stephen_Eades_polymodel_selection.ipynb', 'repo_name': 'stephen-eades/Machine-learning-exercises', 'size': '8847'}