<jupyter_start><jupyter_text><NAME> - G10   Practical Work 1 For this practical work, the student will have to develop a Python program that is able to implement the gradient descent in order to achieve the linear regression of a set of datapoints.  Import numpy, matplotlib.pyplot and make it inline<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.metrics import r2_score<jupyter_output><empty_output><jupyter_text>Read RegData csv file into numpy array  (check your data) Data sourcehttps://college.cengage.com/mathematics/brase/understandable_statistics/7e/students/datasets/slr/frames/frame.html<jupyter_code>train_data = pd.read_csv("RegData.csv", header=None)
train_data = train_data.values<jupyter_output><empty_output><jupyter_text>Explore your data<jupyter_code>print(train_data)<jupyter_output>[[2.9000001  4.        ]
 [6.69999981 7.4000001 ]
 [4.9000001  5.        ]
 [7.9000001  7.19999981]
 [9.80000019 7.9000001 ]
 [6.9000001  6.0999999 ]
 [6.0999999  6.        ]
 [6.19999981 5.80000019]
 [6.         5.19999981]
 [5.0999999  4.19999981]
 [4.69999981 4.        ]
 [4.4000001  4.4000001 ]
 [5.80000019 5.19999981]]<jupyter_text>Define variables X and y. Assign first column data to X and second column to yNote: X is the independent variable (input to LR model) and y is the dependent variable (output)<jupyter_code>x_input = train_data[:,0]
y_output = train_data[:,1]<jupyter_output><empty_output><jupyter_text>Explore your data<jupyter_code>x_input
y_output<jupyter_output><empty_output><jupyter_text>Plot the original data (scatter plot of X,y)<jupyter_code>plt.scatter(x_input, y_output)
plt.show()<jupyter_output><empty_output><jupyter_text>LR Full Implementation  Step1: Initialize parameters (theta_0 & theta_1) with random value or simply zero. Also choose the Learning rate.  ![image.png](attachment:image.png)<jupyter_code>theta_0, theta_1, alpha = 0, 0, 0.01<jupyter_output><empty_output><jupyter_text>Step2: Use (theta_0 & theta_1) to predict the output h(x)= theta_0 + theta_1 * x.![image.png](attachment:image.png) Note: you will need to iterate through all data points<jupyter_code>hypothesis = theta_0 + theta_1 * x_input
hypothesis<jupyter_output><empty_output><jupyter_text>Step3: Calculate Cost function 𝑱(theta_0,theta_1 ).![image.png](attachment:image.png)![image-2.png](attachment:image-2.png)<jupyter_code>m = len(x_input)
cost_func = (1/(2 * m)) * ((np.linalg.norm(hypothesis - y_output)) ** 2)
print(cost_func)<jupyter_output>16.305384479692314<jupyter_text>Step4: Calculate the gradient.![image.png](attachment:image.png)![image-2.png](attachment:image-2.png)<jupyter_code>gradients = [(1/m) * np.sum(hypothesis - y_output), (1/m) * np.sum(x_input.dot(hypothesis - y_output))]
gradients<jupyter_output><empty_output><jupyter_text>Step5: Update the parameters (simultaneously).![image.png](attachment:image.png)![image-2.png](attachment:image-2.png)<jupyter_code>theta_0 = theta_0 - alpha * gradients[0]
theta_1 = theta_1 - alpha * gradients[1]
theta_0
theta_1<jupyter_output><empty_output><jupyter_text>Step6: Repeat from 2 to 5 until converge to the minimum or achieve maximum iterations.![image.png](attachment:image.png)<jupyter_code>costs = []
 for i in range(1000):
    costs.append(cost_func)
    hypothesis = theta_0 + theta_1 * x_input
    cost_func = (1/(2 * m)) * ((np.linalg.norm(hypothesis - y_output)) ** 2)
    gradients = [(1/m) * np.sum(hypothesis - y_output), (1/m) * np.sum(x_input.dot(hypothesis - y_output))]
    theta_0 = theta_0 - alpha * gradients[0]
    theta_1 = theta_1 - alpha * gradients[1]
hypothesis<jupyter_output><empty_output><jupyter_text>Predict y values using the LR equation h(x)= theta_0 + theta_1 * x<jupyter_code>y_predict = theta_0 + theta_1 * x_input<jupyter_output><empty_output><jupyter_text>Plot  LR equation output (fitted line) with the original data (scatter plot of X,y)<jupyter_code>plt.scatter(x_input, y_output)
plt.plot(x_input, y_predict, 'r')
plt.show()<jupyter_output><empty_output><jupyter_text>Use R2 score to evaluate LR equation output![image.png](attachment:image.png)![image-2.png](attachment:image-2.png)![image-3.png](attachment:image-3.png)https://en.wikipedia.org/wiki/Coefficient_of_determination<jupyter_code>r2_score(y_output, hypothesis)<jupyter_output><empty_output><jupyter_text>GD vectorize Implementation Implement GD without iterate through data points i.e. use vector operations<jupyter_code>x_input = train_data[:,0]
y_output = train_data[:,1]
y_output = y_output.reshape(13,1)
x_input_0 = np.ones_like(y_output)
x_input = np.column_stack((x_input_0, x_input))
x_input
y_output
theta = np.zeros((2, 1)) 
alpha = 0.01
m = len(x_input)<jupyter_output><empty_output><jupyter_text>Plot the output and calculate R2 score Make sure that you obtained the same results<jupyter_code>costs = [] 
for i in range(1000):
    hypothesis = x_input@theta 
    print(hypothesis)
    cost_func = (1/(2 * m)) * ((np.linalg.norm(hypothesis - y_output)) ** 2)
    print("------------")
    costs.append(cost_func)
    gradients = (1/m) * (x_input.T@(hypothesis - y_output))
    theta = theta - alpha * gradients
    print(theta)
    print("************")
hypothesis
r2_score(y_output, hypothesis)<jupyter_output><empty_output><jupyter_text>Plot loss function Repeat your last vectorized implementaion version and save loss for each iteration (epoch)  Plot loss vs. iterations<jupyter_code>plt.plot(np.linspace(1, 10, num = 10),costs[:10])
plt.xlabel("iteration")
plt.ylabel("MSE Loss")
plt.show()<jupyter_output><empty_output><jupyter_text>Multivariate LR  Read MultipleLR csv file into numpy array  (check your data) Data sourcehttps://college.cengage.com/mathematics/brase/understandable_statistics/7e/students/datasets/slr/frames/frame.html<jupyter_code>train_data_multiple = pd.read_csv("MultipleLR.csv", header=None)
train_data_multiple = train_data_multiple.values
y_output = train_data_multiple[:,3]
y_output = y_output.reshape(25,1)
x_input_0 = np.ones_like(y_output)
x_input = np.column_stack((x_input_0, train_data_multiple[:, :3]))
x_input
y_output<jupyter_output><empty_output><jupyter_text>Repeat your implementation but for more than one variable<jupyter_code>theta = np.array([[0], [0], [0], [0]])
alpha = 0.0001<jupyter_output><empty_output><jupyter_text>Predict y values using the LR equation h(x)= theta_0 + theta_1 * x1 + theta_2 * x2 + theta_3 * x3<jupyter_code>hypothesis = x_input@theta<jupyter_output><empty_output><jupyter_text>Plot loss vs. iterations<jupyter_code>m = len(x_input)
costs = [] 
for i in range(100):
    hypothesis = x_input@theta 
    print(hypothesis)
    cost_func = (1/(2 * m)) * ((np.linalg.norm(hypothesis - y_output)) ** 2)
    print("------------")
    costs.append(cost_func)
    gradients = (1/m) * (x_input.T@(hypothesis - y_output))
    theta = theta - alpha * gradients
    print(theta)
    print("************")

hypothesis<jupyter_output><empty_output><jupyter_text>Use R2 score to evaluate LR equation output<jupyter_code>r2_score(y_output, hypothesis)
plt.plot(np.linspace(1, 100, num = 100),costs)
plt.xlabel("iteration")
plt.ylabel("MSE Loss")
plt.show()<jupyter_output><empty_output><jupyter_text>Bonus LR Using sklearn  Single Variable  Build a LR model usin linearmodel.LinearRegression() from sklearn library<jupyter_code>from sklearn.linear_model import LinearRegression
train_data = pd.read_csv("RegData.csv", header=None)
train_data = train_data.values
x_input = train_data[:,0]
x_input = x_input.reshape(-1, 1)
y_output = train_data[:,1]<jupyter_output><empty_output><jupyter_text>Train the model (fit the model to the training data)<jupyter_code>reg_model = LinearRegression()
reg_model.fit(x_input, y_output)<jupyter_output><empty_output><jupyter_text>Predict y values using the trained model<jupyter_code>y_predict = reg_model.predict(x_input)<jupyter_output><empty_output><jupyter_text>Plot model output (fitted line) with the original data (scatter plot of X,y)<jupyter_code>plt.scatter(x_input, y_output)
plt.plot(x_input, y_predict, 'r')
plt.show()<jupyter_output><empty_output><jupyter_text>Use R2 score to evaluate model output<jupyter_code>reg_model.score(x_input, y_output)
r2_score(y_output, y_predict)<jupyter_output><empty_output><jupyter_text>Repeat for Mulivariate<jupyter_code>train_data_multiple = pd.read_csv("MultipleLR.csv", header=None)
train_data_multiple = train_data_multiple.values
y_output = train_data_multiple[:,3]
y_output = y_output.reshape(25,1)
x_input_0 = np.ones_like(y_output)
x_input = np.column_stack((x_input_0, train_data_multiple[:, :3]))
reg_model = LinearRegression()
reg_model.fit(x_input, y_output)
y_predict = reg_model.predict(x_input)
reg_model.score(x_input, y_output)
r2_score(y_output, y_predict)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1281001137656428, 'fraction_numerical': 0.04106939704209329, 'mean_word_length': 6.578448275862069, 'pattern_counts': {'":': 0, '<': 116, '<?xml version=': 0, '>': 116, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '29', 'dir': 'jupyter-structured-clean-dedup', 'id': '644441', 'license': 'MIT', 'n_tokens_mistral': 3522, 'n_tokens_neox': 3251, 'n_words': 622, 'path': 'Gradient descent in order to achieve the linear regression.ipynb', 'repo_name': 'mohamedalifaragitiai/Optimization-for-Machne-Learning', 'size': '8801'}