<jupyter_start><jupyter_text>Encoding (Label encoding and One-hot encoding)  <NAME> (BS17B011) These are simple exercises useful for notebook 2<jupyter_code>import pandas as pd<jupyter_output><empty_output><jupyter_text>Open 'grades.csv' and use appropriate encoding technique to convert to numerical data<jupyter_code>df = pd.read_csv("files/grades.csv")
df["Grades"].unique()<jupyter_output><empty_output><jupyter_text>As we can see, there are 6 unique grades - S, A, B, C, E, FAccording to IITMs grading system, the grades are S, A, B, C, D, E, F: ordered from the first priority (1) to the last priority (7) - based on rank 1 to 7<jupyter_code># write a function to do Label encoding
def label_encoding(column):
    grades = {"S":1,"A":2,"B":3,"C":4,"D":5,"E":6,"F":7}
    n = len(column)
    Labeled_data = pd.Series([0]*10,name="Grades")
    for i in range(n):
        Labeled_data[i] = grades[column[i]]
    return Labeled_data
grades_labeled = label_encoding(df["Grades"])
df["Grades"] = grades_labeled
#Dataframe with grades label encoded
df<jupyter_output><empty_output><jupyter_text>Open 'colors.csv' and use appropriate encoding technique to convert to numerical data<jupyter_code>df = pd.read_csv("files/color.csv")<jupyter_output><empty_output><jupyter_text>Hint : you can achieve using pandas (no need for sklearn) check the documentation<jupyter_code># One-hot encoding features - object and colors
df_one_hot = pd.concat([pd.get_dummies(df.Objects),pd.get_dummies(df.Color)],axis=1)
#Dataframe with Object and Color one-hot encoded
print("Numerically encoded data")
df_one_hot<jupyter_output>Numerically encoded data<jupyter_text>Answer the following for both the data - What are the categories in the data?- what is the relationship between different categories in the grades column?- Which encoding to be used Label encoding or one-hot encoding?Note: If there is need of any assumption on grades assume grades follow pattern as it is in IIT M There is one ordinal categorical feature in the "grades.csv" data, which is:- Grades: It specifies the grade acquired by each individual, which is represented by a single letter (S,A,B,C,D,E,F)There are two nominal categorical features in the "colors.csv" data - Objects: It specifies an object (eg: Leaves, Blood, Sea, etc.)- Color: It specifies the color of the object (eg: Green, Red, Blue, etc.) The "Grades" column in the "grades.csv" represents ordinal categorical feature. The grades S, A, B, C, E, F has clear ordering where any one grade is either better or worse than another. The highest to lowest ordering of grades is as follows:S>A>B>C>D>E>F For categorical variables in dataset "grades.csv", label encoding is preferred, as grades represent oordinal data and there is an implicit ordering of the different categories of grades (S>A>B>C>D>E>F). On the other hand, the features "Object", "Color", of "color.csv" are nominal variables that do not have any implicit ordering of the different categories. Hence, one-hot encoding is more suitable in this case, where every unique value in the feature, is a added as an additional binary feature in the encoded data.  Regularized Least Squares  Ridge Regression $$\hat{\theta} = argmin_{\theta}((y - X\theta)^\top(y - X\theta) + \lambda \theta^\top \theta)$$ Complete the following functions for ridge rigression using gradient descent (without sklearn)  - You can reuse the functions you have written in last week- Also you need to change gradient and cost accordinglyHint: If you use numpy for gradient calculation for theta, you may have to flatten the numpy array<jupyter_code>def computeCost(X, y, theta,lamda):
    """
    Compute cost for linear regression. Computes the cost of using theta as the
    parameter for linear regression to fit the data points in X and y.
    
    Parameters
    ----------
    X : with n features
    
    y : with (len(X),1)
    theta : array_like -> The parameters for the regression function. This is a vector of 
        shape (n+1, 1).
    
    Returns
    -------
    J : float -> The value of the regression cost function.
    
    """
    # You need to return the following variables correctly
    m = X.shape[0]
    #Cost function expression - Each term is divided by 2m to improve stability
    J = (np.sum((np.dot(X,theta)-y)**2))/(2*m)+(lamda*np.dot(theta.T,theta))/(2*m)
    
    return J
def gradientDescent(X_appended, y, theta, alpha, lamda=10,max_iters= 1000):
    """
    Performs gradient descent to learn `theta`. Updates theta by taking `num_iters`
    gradient steps with learning rate `alpha`.
    
    Parameters
    ----------
    X : with n degree polynomial features 
    
    y : array with shape of (len(X), 1)
    
    theta : array with shape of (n+1,1)
    
    alpha : float value, called as "learning rate"
    
    max_iters: maximum no of iteration
    
    Returns
    -------
    theta : array wih shape of (n,1),The learned linear regression parameters
    
    J_history : A python list for the values of the cost function after every iteration. This is to check for convergence
    
    
    Hint: 
    ------------
    1. Peform a single gradient step on the parameter vector theta.
    2. Loop over the number of iterations to update step by step.
    """
    # Initialize some useful values
    J_history = [np.inf]
    count = 0
    cost = 0
    m = X_appended.shape[0]
    
    #Derivative of cost function
    diff_cost = lambda y_cap,y,x,theta,lamda,m: (1/m)*(np.dot((y_cap - y).T,x)).T + (1/m)*lamda*theta
    precision = 10**(-6)
    while abs(J_history[-1]-cost)>precision:
        if J_history[0]==np.inf:
            del J_history[0]
        J_history.append(cost)
        y_pred = np.dot(X_appended,theta)
        #Calculating Cost function
        cost = computeCost(X_appended, y, theta,lamda)
        #Updating Parameters
        theta -= alpha*diff_cost(y_pred,y,X_appended,theta,lamda,m)
        #Number of iterations
        count += 1
    
    del J_history[0]
    J_history.append(cost)
    
    return theta, J_history<jupyter_output><empty_output><jupyter_text>Load "dataWeek4Train.npz" for training the model and "dataWeek4Test.npz" for validating ridge rigressionNote: use np.load()<jupyter_code>import numpy as np
from matplotlib import pyplot as plt
#importing training data
data_train = np.load("files/dataWeek4Train.npz")
#importing test data
data_test = np.load("files/dataWeek4Test.npz")<jupyter_output><empty_output><jupyter_text>Using functions written above try to fit the data with polynomial of degree 3 and  - plot the fit on train data, for $\lambda$ = [0,100,10000] - take learning rate (here it is alpha) 0.0001, - max_iters as 1000 (you can change if you wish)- Write comment how the fit changes as lambda and what is the reason?<jupyter_code># Write you code Here
X_train = data_train["X_train"].reshape(-1,1)   #training feature data
y_train = data_train["y_train"].reshape(-1,1)   #training labels

X_test = data_test["X_test"].reshape(-1,1)      #test feature data
y_test = data_test["y_test"].reshape(-1,1)      #test labels   

m = len(X_train)

#Transforming into cubic polynomial
X_train_app = np.append(np.ones((m,1)),X_train,axis=1)
x_plot = np.linspace(min(X_train),max(X_train),1000).reshape(-1,1)
x_plot = np.append(np.ones((1000,1)),x_plot,axis = 1)
for i in range(2,4):
    X_train_app = np.append(X_train_app,X_train_app[:,1].reshape(-1,1)**i,axis=1)
    x_plot = np.append(x_plot,x_plot[:,1].reshape(-1,1)**i,axis=1)
#Training model with train data
lam_values = [0,100,10000]
alpha = 0.0001
theta_list = []
for lamda in lam_values:
    #Training model
    theta = np.array([0]*4).reshape(-1,1).astype(float)
    theta, J_history = gradientDescent(X_train_app, y_train, theta, alpha, lamda, max_iters = 10000)
    theta_list.append(theta)
    #Calculating mse
    m = X_train.shape[0]
    mse = (np.sum(np.dot(X_train_app,theta)-y_train)**2)/m
    
    #Plotting
    plt.figure()
    plt.plot(X_train,y_train,"o")
    plt.plot(x_plot[:,1],np.dot(x_plot,theta),"-")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.legend(["Training data","Cubic Fit"])
    plt.title("Cubic Fit with ridge regression parameter lambda = "+str(lamda)+"; Cost =  "+str(J_history[-1])+"; MSE = %f"%mse)
#Validating with test/validation data
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(3)

X_test_app = poly.fit_transform(X_test)
x_plot_test = np.linspace(min(X_test),max(X_test),1000).reshape(-1,1)
x_plot_test = poly.fit_transform(x_plot_test)
m_test = X_test.shape[0]
for i in range(3):
    plt.figure()
    plt.plot(X_test,y_test,"o")
    y_pred = np.dot(X_test_app,theta_list[i])
    plt.plot(x_plot_test[:,1],np.dot(x_plot_test,theta_list[i]),"-")
    plt.legend(["Test data","Cubic Fit"])
    mse = np.sum((y_pred-y_test)**2)/m_test
    plt.xlabel("x")
    plt.ylabel("y")
    plt.title("Test fit; lambda = "+str(lam_values[i])+"; mse = %f"%mse)<jupyter_output><empty_output><jupyter_text>Use same data "dataWeek4Train.npz" for training the model and "dataWeek4Test.npz" for validating the model for all three models4In the following exercises you can use inbuilt regressionn functions from sklearn<jupyter_code>#Training with sklearn Ridge Regression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge

poly = PolynomialFeatures(3)
X_train_poly = poly.fit_transform(X_train)

theta_sk_list = []
for lamda in lam_values:
    #Training model using sci-kit learn's ridge regression
    clf = Ridge(alpha=lamda)
    clf.fit(X_train_poly[:,1:],y_train)
    #Calculating MSE
    y_pred = clf.predict(X_train_poly[:,1:])
    m = len(X_train)
    mse = np.sum((y_pred-y_train)**2)/m
    #Plotting Training data fits
    plt.figure()
    plt.plot(X_train,y_train,"o")
    plt.plot(x_plot[:,1],clf.predict(x_plot[:,1:]),"-")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.legend(["Trainig data","Cubic Fit"])
    theta_sk = np.append(clf.intercept_,clf.coef_).reshape(-1,1)
    theta_sk_list.append(theta_sk)
    plt.title("Cubic Fit with ridge regression parameter lambda = "+str(lamda)+"; MSE =  "+str(mse))
#validating sklearn model with test/validation data
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(3)

X_test_app = poly.fit_transform(X_test)
x_plot_test = np.linspace(min(X_test),max(X_test),1000).reshape(-1,1)
x_plot_test = poly.fit_transform(x_plot_test)
m_test = X_test.shape[0]
for i in range(3):
    plt.figure()
    plt.plot(X_test,y_test,"o")
    y_pred = np.dot(X_test_app,theta_sk_list[i])
    plt.plot(x_plot_test[:,1],np.dot(x_plot_test,theta_sk_list[i]),"-")
    plt.legend(["Test data","Cubic Fit"])
    mse = np.sum((y_pred-y_test)**2)/m_test
    plt.xlabel("x")
    plt.ylabel("y")
    plt.title("Test fit; lambda = "+str(lam_values[i])+"; mse = %f"%mse)
    
#Coefficients with manual ridge regression
print("Without sklearn - Manual Code\n")
for i in range(3):
    print("Estimated parameters for lambda = " + str(lam_values[i]))
    print(theta_list[i],end= "\n\n")
#Coefficients with sklearn ridge regression
print("Using sklearn\n")
for i in range(3):
    print("Estimated parameters for lambda = " + str(lam_values[i]))
    print(theta_sk_list[i],end= "\n\n")<jupyter_output>Using sklearn

Estimated parameters for lambda = 0
[[-0.76127513]
 [-0.46791175]
 [ 1.06741131]
 [ 0.01839844]]

Estimated parameters for lambda = 100
[[ 0.07642626]
 [-0.0691705 ]
 [ 1.00241723]
 [ 0.00648833]]

Estimated parameters for lambda = 10000
[[ 1.11581176e+01]
 [-6.92409899e-04]
 [ 1.42630833e-01]
 [ 4.03167672e-03]]<jupyter_text>The values of the predicted parameters determined by sklearn and manual code are almost the same for all values of lambda. Except when lambda = 10000, the bias term and the coefficient of the quadratic term predicted using sklearn are not very close to those predicted using manual code. **As the value of lambda increases, the value of mean-squared error (MSE) increases, thereby resulting in poorer fitting of training data (underfitting in extreme cases - high lambda). Higher values of lambda force the optimisation algorithm to minimise the regularising term, which results in decaying of parameters, leading to more simpler models**   Linear Regression (Recap) Write a generic function decribed in the following cell which takes in data, regression type - lasso,ridge and usual linear , degree of the polynomial and then the alpha values (which are regularization parameters -lambda $\lambda$ , in sklearn they are named as alpha so its just a notaitonal difference).Hint: Use PolynomialFeatures from sklearn and set bias to Flase Use 9 degree polynomial features to fit the data using Linear Rigression. Comment on the fit of the model (Overfit, underfit, perfect fit) and also plot prediction on training data along with ground truth<jupyter_code>''' Import neccessay packages '''
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.metrics import mean_squared_error

def regression(X,Y,reg_type ="ridge",degree = 9,alpha=0.1 ):
    ''' 
    Write a function which takes data X,Y
    Type of regression - usual linear , ridge , lasso
    Degree of polynomial features - here it is 10
    Regularization parameter alpha 
    Return predicted output Y hat, Mean square error , regression object
    '''
    poly = PolynomialFeatures(degree,include_bias = False)
    X_poly = poly.fit_transform(X)
    m = X.shape[0]
    
    if reg_type == "linear":  #Usual Linear Regression
        reg = LinearRegression()
        reg.fit(X_poly,Y)
        Y_hat = reg.predict(X_poly).reshape(-1,1)
        mse = np.sum((Y_hat-Y)**2)/m
        
    elif reg_type == "ridge":   #Ridge Regression
        reg = Ridge(alpha=alpha,max_iter=10**6) 
        reg.fit(X_poly,Y)
        Y_hat = reg.predict(X_poly).reshape(-1,1)
        mse = np.sum((Y_hat-Y)**2)/m
        
    elif reg_type == "lasso":   #Lasso Regression
        reg = Lasso(alpha=alpha,max_iter=10**6)
        reg.fit(X_poly,Y)
        Y_hat = reg.predict(X_poly).reshape(-1,1)
        mse = np.sum((Y_hat-Y)**2)/m
    else:      #Default
        raise("Not Valid")
    return Y_hat, mse , reg

# Plotting and calling the function
Y_hat, mse_train , reg = regression(X_train,y_train,reg_type ="linear",degree = 9)


#Plotting fit with training dataset
plt.figure()
plt.plot(X_train,y_train,"o")
x_poly_train = np.linspace(min(X_train),max(X_train))
poly = PolynomialFeatures(9,include_bias = False)
x_poly_train = poly.fit_transform(x_poly_train)
plt.plot(x_poly_train[:,0],reg.predict(x_poly_train),"-")
plt.title("Training data degree 9 polynomial fit (Linear Regression); mse = %f"%mse_train)
plt.xlabel("x")
plt.xlabel("y")
plt.legend(["Training data","Degree 9 fit"])


#Plotting fit with test dataset
poly = PolynomialFeatures(9,include_bias = False)
X_test_app = poly.fit_transform(X_test)
plt.figure()
plt.plot(X_test,y_test,"o")
x_poly_test = np.linspace(min(X_test),max(X_test))
poly = PolynomialFeatures(9,include_bias = False)
x_poly_test = poly.fit_transform(x_poly_test)
mse_test = np.sum((reg.predict(X_test_app)-y_test)**2)/X_test.shape[0]
plt.plot(x_poly_test[:,0],reg.predict(x_poly_test),"-")
plt.title("Test data degree 9 polynomial fit (Linear Regression); mse = %f"%mse_test)
plt.xlabel("x")
plt.xlabel("y")
plt.legend(["Test data","Degree 9 fit"])<jupyter_output><empty_output><jupyter_text>**As can be seen above, the model overfits the data as the training error 0, with all training data points passing through the curve, while the test/validation mse is very high.**  Ridge Rigression and Lasso for polynomial fitting with degree 9In previous you observed that how the model has fit the training data, now try the same thing using regularization both Ridge and Lasso. Use the generic function that you wrote in previous cell. - Now use polynomial regression of order 9 along with Ridge regression and Lasso regression on "dataWeek4Train.npz" for $\lambda= [0, 1, 10^5, 10^{10}]$ and plot the true vs predicted values for each $\lambda$ (Note here that in sklearn lambda is alpha)- Plot the prediction on training data along with ground truth for various values of $\lambda$ for both ridge and lasso. And also print the coefficients of model for each $\lambda$- Also plot trainning error vs $\lambda$Note : Here you can use ridge Rigression from sklearn and also set bias to False in polynomial features  - You can use library to generate polynomial features  Lasso Regression$$\hat{\theta} = argmin_{\theta}(\frac{1}{2} \sum_{i=1}^{N} (y_i-\theta_0-\sum_{j=1}^p x_{ij}\theta_j)^2 + \lambda \sum_{j=1}^{p}|\theta_j|)$$<jupyter_code>#Using Lasso Regression to training dataset
lam_vals = [0,1,10**5,10**10]
mse_lasso = []

lasso_list = []

for lamda in lam_vals:
    Y_hat, mse_train , reg_lasso = regression(X_train,y_train,reg_type ="lasso",degree = 9,alpha=lamda)
    mse_lasso.append(mse_train)
    lasso_list.append(reg_lasso)
    
    x_poly_train = np.linspace(min(X_train),max(X_train))
    x_poly_train = poly.fit_transform(x_poly_train)
    
    #Plots
    plt.figure()
    plt.plot(X_train,y_train,"o")
    plt.plot(x_poly_train[:,0],reg_lasso.predict(x_poly_train),"-")
    plt.title("Training data degree 9 polynomial fit (Lasso Regression); lambda = %d"%lamda + "; mse = %f"%mse_train)
    plt.xlabel("x")
    plt.ylabel("y")
    plt.legend(["Training data","Degree 9 fit"])
#Using Ridge Regression to training dataset
mse_ridge = []  
ridge_list = []
for lamda in lam_vals:
    Y_hat, mse_train , reg_ridge = regression(X_train,y_train,reg_type ="ridge",degree = 9,alpha=lamda)
    mse_ridge.append(mse_train)
    ridge_list.append(reg_ridge)
    
    x_poly_train = np.linspace(min(X_train),max(X_train))
    x_poly_train = poly.fit_transform(x_poly_train)
    
    plt.figure()
    plt.plot(X_train,y_train,"o")
    plt.plot(x_poly_train[:,0],reg_ridge.predict(x_poly_train),"-")
    plt.title("Training data degree 9 polynomial fit (Ridge Regression); mse = %f"%mse_train +" ;lambda = %d"%lamda)
    plt.xlabel("x")
    plt.ylabel("y")
    plt.legend(["Training data","Degree 9 fit"])<jupyter_output>C:\Users\Dell\anaconda3\lib\site-packages\sklearn\linear_model\_ridge.py:188: LinAlgWarning: Ill-conditioned matrix (rcond=1.04135e-17): result may not be accurate.
  overwrite_a=False)<jupyter_text>Hence, increase in lambda results in simpler models in both Lasso and Ridge regressionNOTE: In both Lasso and Ridge regression, training the model with lambda = 0 is as good as simple linear regression. However, using sklearn's lasso regression with lambda = 0 is not recommended as per the warning<jupyter_code>#Generating Training error vs log(lambda+1) plot - Lasso Regression
plt.plot(np.log10(np.array(lam_vals)+1),mse_lasso,"-o")
plt.xlabel("log(lambda+1)")
plt.ylabel("Training MSE")
plt.title("Training MSE vs log(Lambda+1) : Lasso")
#Generating Training error vs log(lambda+1) plot - Ridge Regression
plt.plot(np.log10(np.array(lam_vals)+1),mse_ridge,"-o")
plt.xlabel("log(lambda+1)")
plt.ylabel("Training MSE")
plt.title("Training MSE vs log(Lambda+1) : Ridge")<jupyter_output><empty_output><jupyter_text>Ridge Rigression and Lasso for polynomial fitting to predict on validation setNow use "dataWeek4Test.npz" to validate the model for same values of $\lambda$s ($\lambda= [0, 1, 10^5, 10^{10}]$) used in previous case. And plot $\lambda$ vs validation error.  - Note : No need to plot the prediction on validation set since the validation points are very few- Print the learned coefficients for Lasso and Ridge rigression and write your observationsHint: Think of the property of Lasso Rigression (for writing observations)<jupyter_code>#Generating Test error vs log(lambda+1) plot - Lasso Regression
test_mse_lasso = []
plt.figure()
for i in range(len(lam_vals)):
    test_mse_lasso.append(np.sum((lasso_list[i].predict(X_test_app)-y_test)**2)/X_test.shape[0])
plt.figure()
plt.plot(np.log10(np.array(lam_vals)+1),test_mse_lasso,"-o")
plt.xlabel("log(lambda+1)")
plt.ylabel("Test MSE")
plt.title("Test MSE vs log(Lambda+1) - Lasso Regression")
#Generating Test error vs log(lambda+1) plot - Ridge Regression
test_mse_ridge = []
for i in range(len(lam_vals)):
    test_mse_ridge.append(np.sum((ridge_list[i].predict(X_test_app)-y_test)**2)/X_test.shape[0])
plt.plot(np.log10(np.array(lam_vals)+1),test_mse_ridge,"-o")
plt.xlabel("log(lambda+1)")
plt.ylabel("Test MSE")
plt.title("Test MSE vs log(Lambda+1) - Ridge Regression")
#Printing the learned coefficients for Lasso Regression
for i in range(len(lam_vals)):
    print("Lambda = %d"%lam_vals[i])
    print("Coefficients: ")
    print(lasso_list[i].coef_)
    print("Intercept: ")
    print(lasso_list[i].intercept_,end = "\n\n")
#Printing the learned coefficients for Ridge Regression
for i in range(len(lam_vals)):
    print("Lambda = %d"%lam_vals[i])
    print("Coefficients: ")
    print(ridge_list[i].coef_)
    print("Intercept: ")
    print(ridge_list[i].intercept_,end = "\n\n")<jupyter_output>Lambda = 0
Coefficients: 
[[ 1.55315827e+00  1.60951239e-01  4.42683559e-01  4.86392799e-01
  -4.54839905e-01 -4.55959576e-02  4.68980382e-02  9.10977519e-04
  -9.62072268e-04]]
Intercept: 
[-1.50533287]

Lambda = 1
Coefficients: 
[[ 7.44116435e-01  1.59227497e-01  1.92021948e-01  4.81193528e-01
  -2.78416132e-01 -4.49908395e-02  2.98148846e-02  8.98201036e-04
  -6.17815448e-04]]
Intercept: 
[-1.4779386]

Lambda = 100000
Coefficients: 
[[-7.44481536e-05  2.43355499e-04 -3.10644737e-04  1.58106688e-03
  -8.54353734e-04  9.95073541e-03  6.17908868e-05 -2.56149945e-04
  -9.43202852e-07]]
Intercept: 
[1.33941411]

Lambda = 10000000000
Coefficients: 
[[-8.41944511e-10  1.37336483e-08 -4.10293246e-09  1.60946222e-07
  -1.85729179e-08  1.78835737e-06 -8.15144653e-08  1.94013952e-05
   1.09490809e-07]]
Intercept: 
[4.75301902]
{'fraction_non_alphanumeric': 0.12220545878480318, 'fraction_numerical': 0.03995793901156677, 'mean_word_length': 4.426445050855867, 'pattern_counts': {'":': 10, '<': 50, '<?xml version=': 0, '>': 65, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '321524', 'license': 'MIT', 'n_tokens_mistral': 8070, 'n_tokens_neox': 7446, 'n_words': 2092, 'path': 'Assignment 4a/G Prashant (BS17B011) - Assignment 4a.ipynb', 'repo_name': 'gprashant123/Data-Analytics-Lab', 'size': '21877'}