<jupyter_start><jupyter_text>Linear Regression We will apply linear regression on the lending club cleaned and filtered dataset, that we have already cleaned and filtered considering our persona Rick who is a risk averse. We will use the dataset to predict the interest rate. We will be performing  1. Linear Regression withour regularization 2. Linear Regression with Lasso regularization (L1 Penalty) 3. Linear Regression with Ridge regularization (L2 Penalty) 3. Linear Regression with ElasticNet regularization (L1 and L2 Penalty) Will also plot graphs to see how the hyperparameter tuning increases the model effectivity and prepare a report on the same<jupyter_code># Imports
import pandas as pd
import numpy as np
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression, RidgeCV, LassoCV, ElasticNetCV
from sklearn import metrics
from sklearn.metrics import mean_squared_error, make_scorer
from sklearn.model_selection import KFold
from IPython.display import display
import matplotlib.pyplot as plt
import seaborn as sns

# Definitions
pd.set_option('display.float_format', lambda x: '%.3f' % x)
%matplotlib inline
#njobs = 4<jupyter_output><empty_output><jupyter_text>Get the dataset<jupyter_code>dataset = pd.read_csv('../Data/finaldataset.csv')
dataset.pop('Unnamed: 0')<jupyter_output><empty_output><jupyter_text>Check the info, describe and head on the dataset<jupyter_code>dataset.info()
dataset.describe()
dataset.head()<jupyter_output><empty_output><jupyter_text>The dataset is already cleaned and filtered, so we will directly apply the model and check the performance and error metrics  Train and Test Split<jupyter_code>Y = dataset.pop('int_rate').values
Y
loan_status = dataset.pop('loan_status').values
X = dataset.values
X
dataset_copy =dataset<jupyter_output><empty_output><jupyter_text>Normal split without K-Folds<jupyter_code>X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.3, random_state = 0)<jupyter_output><empty_output><jupyter_text>Defining the method that will Train and Evaluate algorithm and will return the error metrics<jupyter_code>def get_result(model, X_train, X_test, Y_train, Y_test):
    sc = StandardScaler()  
    X_train = sc.fit_transform(X_train)  
    X_test = sc.transform(X_test) 
    model.fit(X_train, Y_train)
    y_pred = model.predict(X_test)
    y_train_pred = model.predict(X_train)
    RMSE = np.sqrt(metrics.mean_squared_error(Y_test, y_pred))
    MAPE_test = np.mean(np.abs((Y_test - y_pred) / Y_test)) * 100
    MAPE_train = np.mean(np.abs((Y_train - y_train_pred) / Y_train)) * 100
    return RMSE, MAPE_test,MAPE_train, y_pred,y_train_pred, model<jupyter_output><empty_output><jupyter_text>1. Linear Regression without regularization<jupyter_code>lr = LinearRegression()
rmse, mape,mape_train, y_pred,y_train_pred, model = get_result(lr,X_train, X_test, Y_train, Y_test)
print("MAPE on Test : ", mape," mape on train data: ",mape_train)
y_train_pred
y_pred
# Plot predictions
plt.scatter(y_train_pred, Y_train, c = "blue", marker = "s", label = "Training data")
plt.scatter(y_pred, Y_test, c = "lightgreen", marker = "s", label = "Validation data")
plt.title("Linear regression")
plt.xlabel("Predicted values")
plt.ylabel("Real values")
plt.legend(loc = "upper left")
#plt.plot([10.5, 13.5], [10.5, 13.5], c = "red")
plt.show()<jupyter_output><empty_output><jupyter_text>The MAPE of our model is 4.6995 which is pretty good. That means our model is able to capture most of the explanatory information. Now we can split the train and test data using the K-Fold and find the average MAPE and check if the performance increases<jupyter_code>columnnames= dataset_copy.columns.values
x = dataset_copy[columnnames]
y = Y
x.shape
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
x= scaler.fit_transform(x)
x_df = pd.DataFrame(x,columns=columnnames)
x_df
import matplotlib.pyplot as plt
import numpy as np

from sklearn.feature_selection import SelectFromModel
from sklearn.linear_model import LassoCV


# We use the base estimator LassoCV since the L1 norm promotes sparsity of features.
clf = LassoCV(cv=5)

# Set a minimum threshold of 0.05
sfm = SelectFromModel(clf, threshold=0.05)
sfm.fit(x_df, y)
n_features = sfm.transform(x_df).shape[1]

# Reset the threshold till the number of features equals two.
# Note that the attribute can be set directly instead of repeatedly
# fitting the metatransformer.
# while n_features > 2:
#     sfm.threshold += 0.1
#     X_transform = sfm.transform(x)
#     n_features = X_transform.shape[1]

# Plot the selected two features from X.
# plt.title(
#     "Features selected from Boston using SelectFromModel with "
#     "threshold %0.3f." % sfm.threshold)
# feature1 = X_transform[:, 0]
# feature2 = X_transform[:, 1]
# plt.plot(feature1, feature2, 'r.')
# plt.xlabel("Feature number 1")
# plt.ylabel("Feature number 2")
# plt.ylim([np.min(feature2), np.max(feature2)])
# plt.show()
x_df[x_df.columns[sfm.get_support(indices=True)]].columns
X_train1, X_test1, Y_train1, Y_test1 = train_test_split(x_df[x_df.columns[sfm.get_support(indices=True)]], Y, test_size = 0.3, random_state = 0)
lr = LinearRegression()
rmse1, mape1,mape_train1, y_pred1,y_train_pred1, model1 = get_result(lr,X_train1, X_test1, Y_train1, Y_test1)
print("MAPE on Test : ", mape1," mape on train data: ",mape_train1)
kf = KFold(n_splits=5)
kf.get_n_splits(X)


MAPE = []
MAPE_Train = []
X_test_array=[]
y_pred_array=[]
Y_test_ret_array = []
Y_train_ret_array = []
y_train_pred_array=[]
lr = LinearRegression()
for train_index, test_index in kf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X[train_index], X[test_index]
    Y_train, Y_test = Y[train_index], Y[test_index]
    rmse, mape,mape_train, y_pred,y_train_pred, model = get_result(lr,X_train, X_test, Y_train, Y_test)
    MAPE.append(mape)
    MAPE_Train.append(mape_train)
    X_test_array.append(X_test)
    y_pred_array.append(y_pred)
    Y_train_ret_array.append(Y_train)
    y_train_pred_array.append(y_train_pred)
#test mapes, calculate for train mapes
MAPE
np.mean(MAPE_Train)<jupyter_output><empty_output><jupyter_text>The K-Fold has split the dataset in some abstract way that we got unacceptable MAPE value in 2 splits. So we will stick to our conventional split of 70:30 and proceed with regularization<jupyter_code>X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.3, random_state = 0)<jupyter_output><empty_output><jupyter_text>2. Linear Regression with Ridge regularization (L2 penalty) Ridge regression is an L2 penalized model where we simply add the squared sum of the weights to our cost function.<jupyter_code># 2* Ridge
ridge = RidgeCV(alphas = [0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6, 10, 30, 60])
ridge.fit(X_train, Y_train)
alpha = ridge.alpha_
print("Best alpha :", alpha)

print("Trying again for more precision with alphas centered around " + str(alpha))
ridge = RidgeCV(alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, alpha * .85, 
                          alpha * .9, alpha * .95, alpha, alpha * 1.05, alpha * 1.1, alpha * 1.15,
                          alpha * 1.25, alpha * 1.3, alpha * 1.35, alpha * 1.4], 
                cv = 10)
ridge.fit(X_train, Y_train)
alpha = ridge.alpha_
print("Best alpha :", alpha)

rmse, mape, y_pred,y_train_pred, model = get_result(ridge,X_train, X_test, Y_train, Y_test)
print("MAPE on Test : ", mape)
# Plot predictions
plt.scatter(y_train_pred, Y_train, c = "blue", marker = "s", label = "Training data")
plt.scatter(y_pred, Y_test, c = "lightgreen", marker = "s", label = "Validation data")
plt.title("Linear regression with Ridge regularization")
plt.xlabel("Predicted values")
plt.ylabel("Real values")
plt.legend(loc = "upper left")
plt.show()
# Plot important coefficients
coefs = pd.Series(ridge.coef_, index = dataset.columns)
print("Ridge picked " + str(sum(coefs != 0)) + " features and eliminated the other " +  \
      str(sum(coefs == 0)) + " features")
imp_coefs = pd.concat([coefs.sort_values().head(10),
                     coefs.sort_values().tail(10)])
imp_coefs.plot(kind = "barh")
plt.title("Coefficients in the Ridge Model")
plt.show()<jupyter_output>Ridge picked 83 features and eliminated the other 1 features<jupyter_text>3. Linear Regression with Lasso regularization (L1 penalty) L1 regularization yields sparse feature vectors : most feature weights will be zero. Sparsity can be useful in practice if we have a high dimensional dataset with many features that are irrelevant.<jupyter_code>lasso = LassoCV(alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1,0.3, 0.6, 1],
               max_iter = 50000, cv = 10)
lasso.fit(X_train,Y_train)
alpha = lasso.alpha_
print("Best alpha :", alpha)

print("Try again for more precision with alphas centered around " + str(alpha))
lasso = LassoCV(alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, 
                          alpha * .85, alpha * .9, alpha * .95, alpha, alpha * 1.05, 
                          alpha * 1.1, alpha * 1.15, alpha * 1.25, alpha * 1.3, alpha * 1.35, 
                          alpha * 1.4], 
                max_iter = 50000, cv = 10)
lasso.fit(X_train, Y_train)
alpha = lasso.alpha_
print("Best alpha :", alpha)
rmse, mape, y_pred,y_train_pred, model = get_result(lasso,X_train, X_test, Y_train, Y_test)
print("MAPE on Test : ", mape)
# Plot predictions
plt.scatter(y_train_pred, Y_train, c = "blue", marker = "s", label = "Training data")
plt.scatter(y_pred, Y_test, c = "lightgreen", marker = "s", label = "Validation data")
plt.title("Linear regression with Lasso regularization")
plt.xlabel("Predicted values")
plt.ylabel("Real values")
plt.legend(loc = "upper left")
plt.show()
# Plot important coefficients
coefs = pd.Series(lasso.coef_, index = dataset.columns)
print("Lasso picked " + str(sum(coefs != 0)) + " features and eliminated the other " +  \
      str(sum(coefs == 0)) + " features")
imp_coefs = pd.concat([coefs.sort_values().head(10),
                     coefs.sort_values().tail(10)])
imp_coefs.plot(kind = "barh")
plt.title("Coefficients in the Lasso Model")
plt.show()<jupyter_output>Lasso picked 79 features and eliminated the other 5 features<jupyter_text>4. Linear Regression with ElasticNet regularization (L1 and L2 penalty) ElasticNet is a compromise between Ridge and Lasso regression. It has a L1 penalty to generate sparsity and a L2 penalty to overcome some of the limitations of Lasso<jupyter_code># 4* ElasticNet
elasticNet = ElasticNetCV(l1_ratio = [0.1, 0.3, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9, 0.95, 1],
                          alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 
                                    0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6], 
                          max_iter = 50000, cv = 10)
elasticNet.fit(X_train, Y_train)
alpha = elasticNet.alpha_
ratio = elasticNet.l1_ratio_
print("Best l1_ratio :", ratio)
print("Best alpha :", alpha )

print("Try again for more precision with l1_ratio centered around " + str(ratio))
elasticNet = ElasticNetCV(l1_ratio = [ratio * .85, ratio * .9, ratio * .95, ratio, ratio * 1.05, ratio * 1.1, ratio * 1.15],
                          alphas = [0.0001, 0.0003, 0.0006, 0.001, 0.003, 0.006, 0.01, 0.03, 0.06, 0.1, 0.3, 0.6, 1, 3, 6], 
                          max_iter = 50000, cv = 10)
elasticNet.fit(X_train, Y_train)
if (elasticNet.l1_ratio_ > 1):
    elasticNet.l1_ratio_ = 1    
alpha = elasticNet.alpha_
ratio = elasticNet.l1_ratio_
print("Best l1_ratio :", ratio)
print("Best alpha :", alpha )

print("Now try again for more precision on alpha, with l1_ratio fixed at " + str(ratio) + 
      " and alpha centered around " + str(alpha))
elasticNet = ElasticNetCV(l1_ratio = ratio,
                          alphas = [alpha * .6, alpha * .65, alpha * .7, alpha * .75, alpha * .8, alpha * .85, alpha * .9, 
                                    alpha * .95, alpha, alpha * 1.05, alpha * 1.1, alpha * 1.15, alpha * 1.25, alpha * 1.3, 
                                    alpha * 1.35, alpha * 1.4], 
                          max_iter = 50000, cv = 10)
elasticNet.fit(X_train, Y_train)
if (elasticNet.l1_ratio_ > 1):
    elasticNet.l1_ratio_ = 1    
alpha = elasticNet.alpha_
ratio = elasticNet.l1_ratio_
print("Best l1_ratio :", ratio)
print("Best alpha :", alpha )

#elasticNet
rmse, mape, y_pred,y_train_pred, model = get_result(elasticNet,X_train, X_test, Y_train, Y_test)
print("MAPE on Test : ", mape)
# Plot predictions
plt.scatter(Y_train, y_train_pred, c = "blue", marker = "s", label = "Training data")
plt.scatter(Y_test, y_pred, c = "lightgreen", marker = "s", label = "Validation data")
plt.title("Linear regression with ElasticNet regularization")
plt.xlabel("Predicted values")
plt.ylabel("Real values")
plt.legend(loc = "upper left")
plt.show()
# Plot important coefficients
coefs = pd.Series(elasticNet.coef_, index = dataset.columns)
print("ElasticNet picked " + str(sum(coefs != 0)) + " features and eliminated the other " +  str(sum(coefs == 0)) + " features")
imp_coefs = pd.concat([coefs.sort_values().head(10),
                     coefs.sort_values().tail(10)])
imp_coefs.plot(kind = "barh")
plt.title("Coefficients in the ElasticNet Model")
plt.show()<jupyter_output>ElasticNet picked 80 features and eliminated the other 4 features
{'fraction_non_alphanumeric': 0.11978585768458622, 'fraction_numerical': 0.03479812625474013, 'mean_word_length': 4.195056006179993, 'pattern_counts': {'":': 0, '<': 46, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '441875', 'license': 'MIT', 'n_tokens_mistral': 5048, 'n_tokens_neox': 4612, 'n_words': 1316, 'path': 'Dimensionality_reduction/Lending_Club_Linear_Regression-Copy1.ipynb', 'repo_name': 'satwik2663/Machine-Laerning---Intrest-Rate-Prediction-for-Peer-to-peer-Lending-Group', 'size': '13449'}