<jupyter_start><jupyter_text>Linear RegressionThe objective of linear regression is to minimize the cost function$J(\theta) = \frac{1}{2m} \sum_{i=1}^{m} (h_\theta(x^{(i)}) - y^{(i)})^2 $ (vectorial form $J(\theta) = \frac{1}{2m} (X\theta - y)^T (X\theta - y)$)where the hypothesis $h_\theta$ is given by the linear model$$ h_\theta = \theta^T x = \theta_0 + \theta_1 x_1 $$The model's parameters are the $\theta_j$ values. These are the values that need to be adjusted to minimize cost $J(\theta)$.   Create Dataset Here I am going for a univariate case to make visulaization simple.<jupyter_code>import pandas as pd
import numpy as np
df = pd.read_csv('Dataset.csv',index_col=0)
df.head()
import seaborn as sns
sns.heatmap(df.corr())<jupyter_output><empty_output><jupyter_text>Exploratory Data AnalysisWe can see from the above heatmap that our field of interest 'Chance of admit' has a positive correlation with all other fields. The most positive correlation is with the fields GRE score, TOEFL score and CGPA.So here I will create two Linear Regression Models one considering all of the available fields as features and the other being picking up only the three most correlated fields and compare the results.  Algorithm for Linear regressionFinding Theta using the Normal Equations¶It is possible to show that the same optimized parameters found implemenating Gradient Descent can be calculated in an elegant, efficient and closed form, using linear algebra. Specifically:$$\theta = (X^T X)^{-1} X^Ty$$Using this formula does not require any feature scaling, and you will get an exact solution in one calculation: there is no "loop until convergence" like in gradient descent.<jupyter_code>def normalEquation(X, y):
    return np.dot(np.dot(np.linalg.inv(np.dot(X.transpose(), X)), X.transpose()), y)<jupyter_output><empty_output><jupyter_text>Algorithm 1 : Taking all fields in the feature vector<jupyter_code>X = df.drop(['ChanceOfAdmit'],axis=1).values
X
y = df['ChanceOfAdmit'].values.reshape(len(X),1)
y[:5]<jupyter_output><empty_output><jupyter_text>Now we need to find optimal weight matrix W (or theta)$$\theta = (X^T X)^{-1} X^Ty$$<jupyter_code>W = normalEquation(X,y)
W<jupyter_output><empty_output><jupyter_text>Now we calulcate the MEAN Squared Error<jupyter_code>MSE = np.linalg.norm(np.dot(X,W) - y)**2/len(X)
MSE<jupyter_output><empty_output><jupyter_text>Algorithm 2 : Taking top 3 fields which are most positively correlated in the feature vector<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Preparing data for matrix manipulation<jupyter_code>X = df[['GRE Score', 'TOEFL Score', 'CGPA']].values
X[:5]
y = df['ChanceOfAdmit'].values.reshape(len(y),1)
y[:5]
W = normalEquation(X,y)
W
MSE = np.linalg.norm(np.dot(X,W) - y)**2/len(X)
MSE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11858974358974358, 'fraction_numerical': 0.00819088319088319, 'mean_word_length': 6.111392405063291, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '371194', 'license': 'MIT', 'n_tokens_mistral': 951, 'n_tokens_neox': 911, 'n_words': 304, 'path': 'Linear-Regression/Multi-Variate Linear Regression.ipynb', 'repo_name': 'kstv364/Machine-Learning-from-Scratch', 'size': '2808'}