<jupyter_start><jupyter_text>What is Regression? Regression analysis is one of the most important fields in statistics and machine learning. There are many regression methods available. Linear regression is one of them. **Used when output is numeric** Regression searches for relationships among variables. For example, you can observe several employees of some company and try to understand how their salaries depend on the features, such as experience, level of education, role, city they work in, and so on.  Linear Regression Linear regression is probably one of the most important and widely used regression techniques. It’s among the simplest regression methods. One of its main advantages is the ease of interpreting results. When implementing linear regression of some dependent variable 𝑦 on the set of independent variables 𝐱 = (𝑥₁, …, 𝑥ᵣ), where 𝑟 is the number of predictors, you assume a linear relationship between 𝑦 and 𝐱: 𝑦 = 𝛽₀ + 𝛽₁𝑥₁ + ⋯ + 𝛽ᵣ𝑥ᵣ + 𝜀. This equation is the regression equation. 𝛽₀, 𝛽₁, …, 𝛽ᵣ are the regression coefficients, and 𝜀 is the random error.  Simple Linear Regression Simple or single-variate linear regression is the simplest case of linear regression with a single independent variable, 𝐱 = x.  ![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAgAElEQVR4Ae2dB7gV1fXFF713ESH4qNJ7ERARlG5DEUEEpVhRQZoIYsfeYsNeYhQLoqigRpMYosbESGIwWEgQG5a/WGOJWLL/35o38xwuc8u775a5c9f5vsfMnXJmn98Z1j77nDMzgJIIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiIAIiE<KEY>EREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAERKAcBP4f5gCHkszjrgoAAAAASUVORK5CYII=) The estimated regression function (red line) has the equation:$$f(x) = b_o + 𝑏_1x$$ Our goal is to calculate the optimal values of the predicted weights 𝑏₀ and 𝑏₁ that minimize the sum of squared residuals (SSR) and determine the estimated regression function.  Multiple Linear Regression Multiple or multivariate linear regression is a case of linear regression with two or more independent variables.If there are just two independent variables, the estimated regression function is:   $$f(x_1,x_2) = b_0 + b_1x_1 + b_2x_2$$It represents a regression plane in a three-dimensional space. The goal of regression is to determine the values of the weights 𝑏₀, 𝑏₁, and 𝑏₂ such that this plane is as close as possible to the actual responses and yield the minimal SSR. ![](img/lin1.png) Plane   Polynomial Regression You can regard polynomial regression as a generalized case of linear regression. You assume the polynomial dependence between the output and inputs and, consequently, the polynomial estimated regression function.In other words, in addition to linear terms like 𝑏₁𝑥₁, your regression function 𝑓 can include non-linear terms such as 𝑏₂𝑥₁², 𝑏₃𝑥₁³, or even 𝑏₄𝑥₁𝑥₂, 𝑏₅𝑥₁²𝑥₂, and so on.The simplest example of polynomial regression has a single independent variable, and the estimated regression function is a polynomial of degree 2:$$ f(x) = b_o + b_1x + b_2x^2$$- Kernel Trick  Regression Assumptions  LinearityThe simple meaning of linearity is understand from scatter plot which is the straight line that drawn in between points on scatter plot. No HeteroskedasticityThe presence of non-constant variance in the error terms results in heteroskedasticity  **Residual Data**: The residual data of the simple linear regression model is the difference between the observed data of the dependent variable y and the fitted values ŷ.  ` plt.scatter(pred, (pred - actual), c='b')`![](img/residual-plot.png) No Multi-ColinearityMulticollinearity refers to a situation in which two or more explanatory variables in a multiple regression model are highly linearly related.   How to identify this:- Using Pearson-correlation. How to fix this:  - Drop one of the two variables  - Create a function to create a new independent variable using the correlated features and drop the correlated features.  Simple Linear Regression With scikit-learn  Import the Packages<jupyter_code>import numpy as np
from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Provide Data<jupyter_code>from sklearn.datasets import make_regression
X, y = make_regression(n_samples=100, n_features=1, noise=5,random_state=42) # noise defines the distribution deviation from the mean and random state allows us generate the same data like seed
import matplotlib.pyplot as plt
plt.scatter(X, y)<jupyter_output><empty_output><jupyter_text>Create a model & fit<jupyter_code>model = LinearRegression()
model.fit(X, y)<jupyter_output><empty_output><jupyter_text>Get Results<jupyter_code>r_sq = model.score(X, y)
print('Coefficient of determination (R2):', r_sq)
print('intercept:', model.intercept_)
print('slope:', model.coef_)

# function for fitted values ie to predict values
# y = 43.1* X +  0.58<jupyter_output><empty_output><jupyter_text>Predict<jupyter_code>X_test, y_test = make_regression(n_samples=100, n_features=1, noise=20 ,random_state=42)
y_pred = model.predict(X_test)
print('predicted response:', y_pred, sep='\n')
plt.scatter(X, y, color="red")
plt.scatter(X_test, y_pred)
model.score(X, y)
r_t = model.score(X_test, y_test)
print('Coefficient of determination (R2) of test:', r_t)<jupyter_output>Coefficient of determination (R2) of test: 0.8460442632197438<jupyter_text>Multiple Linear Regression With scikit-learn  Provide Data<jupyter_code>from sklearn.datasets import make_regression
X_m, y_m = make_regression(n_samples=100, n_features=2, noise=20,random_state=42)
X_m[1:5]<jupyter_output><empty_output><jupyter_text>Create a model & fit<jupyter_code>model = LinearRegression().fit(X_m, y_m)<jupyter_output><empty_output><jupyter_text>Get Results<jupyter_code>r_sq = model.score(X_m, y_m)
print('coefficient of determination:', r_sq)
print('intercept:', model.intercept_)
print('slope:', model.coef_)<jupyter_output>coefficient of determination: 0.959604849170989
intercept: 0.4327161689219041
slope: [84.26785337 74.15664645]<jupyter_text>Predict<jupyter_code>Xm_test, ym_test = make_regression(n_samples=20, n_features=2, noise=20, random_state=42)

ym_pred = model.predict(Xm_test)
print('predicted response:', ym_pred, sep='\n')
print('Coefficient of determination (R2) of test:', model.score(Xm_test,ym_test))
from sklearn.metrics import r2_score
r2_score(ym_test, ym_pred)<jupyter_output><empty_output><jupyter_text>Cost Function It is a function that measures the performance of a Machine Learning model for given data. Cost Function quantifies the error between predicted values and expected values and presents it in the form of a single real number. $$MSE = \frac{1}{m}\sum_{n=1}^{m} (y_p-y)^{2}$$  Mean Square Error (MSE)<jupyter_code>from sklearn.metrics import mean_squared_error
mean_squared_error(y, y_pred)
np.mean((y - y_pred)**2)<jupyter_output><empty_output><jupyter_text>Mean Square Error (MAE)<jupyter_code>from sklearn.metrics import mean_absolute_error
mean_absolute_error(y,y_pred)<jupyter_output><empty_output><jupyter_text>Adjusted ($R^2$)Adjusted R2 also indicates how well terms fit a curve or line, but adjusts for the number of terms in a model. If you add more and more useless variables to a model, adjusted r-squared will decrease. If you add more useful variables, adjusted r-squared will increase.Adjusted $R^2$ will always be less than or equal to $R^2$.<jupyter_code>def adj_r2 (X,y,model):
    r_squared = model.score(X,y)
    return(1 - (1-r_squared)*(len(y)-1)/(len(y)-X.shape[1]-1))
adj_r2(Xm_test, ym_test, model)<jupyter_output><empty_output><jupyter_text>Real Life Example<jupyter_code>from sklearn.datasets import load_boston
import pandas as pd

Xb,yb =load_boston(return_X_y=True)

df_boston = pd.DataFrame(Xb,columns = load_boston().feature_names)
df_boston.head()
df_boston.info()
df_boston.describe()
df_boston.isna().sum()
# Correlation of the features
import seaborn as sns
corr = df_boston.corr()

plt.figure(figsize=(14, 14))
ax = sns.heatmap(
    corr, 
    vmin=-1, vmax=1, center=0,
    cmap=sns.diverging_palette(20, 220, n=200),
    square=True, annot = True
)
ax.set_xticklabels(
    ax.get_xticklabels(),
    rotation=45,
    horizontalalignment='right'
)
ax.set_ylim(len(corr)+0.5, -0.5);
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(Xb,yb, test_size=0.3, random_state=42)<jupyter_output><empty_output><jupyter_text>`train_test_split` splits arrays or matrices into random train and test subsets. That means that everytime you run it without specifying `random_state`, you will get a different result. Thus, your project will not be reproducible.<jupyter_code>modelb = LinearRegression(normalize=False)
modelb.fit(X_train,y_train)

print("Score of the train set",modelb.score(X_train,y_train))
print("Score of the test set",modelb.score(X_test,y_test))
print("Adj. R2 of the train set",adj_r2(X_train,y_train,modelb))
print("Adj. R2 of the test set",adj_r2(X_test,y_test,modelb))
importance = modelb.coef_
for i in range(len(importance)):
    print("Feature", df_boston.columns[i], "Score:", importance[i])
# Dropping colerated features
new_df = df_boston.drop(["AGE","INDUS"],axis=1)
X_train, X_test, y_train, y_test = train_test_split(new_df,yb, test_size=0.3, random_state=42)
modelb = LinearRegression(normalize=True)
modelb.fit(X_train,y_train)
print("Score of the train set",modelb.score(X_train,y_train))
print("Score of the test set",modelb.score(X_test,y_test))
print("Score of the train set",adj_r2(X_train,y_train,modelb))
print("Score of the test set",adj_r2(X_test,y_test,modelb))
# Scaling
from scipy import stats
import numpy as np
z = np.abs(stats.zscore(df_boston))
z
len(np.where(z > 3)[0])
outliers = list(set(np.where(z > 3)[0]))
new_df = df_boston.drop(outliers,axis = 0).reset_index(drop = False)
display(new_df)

y_new = yb[list(new_df["index"])]
len(y_new)
X_new = new_df.drop('index', axis = 1)

from sklearn.preprocessing import StandardScaler, MinMaxScaler
X_scaled = StandardScaler().fit_transform(X_new)
X_scaled
# Outliers removed & scaling
X_train, X_test, y_train, y_test = train_test_split(X_scaled,y_new, test_size=0.3, random_state=42)
modelb = LinearRegression(normalize=False)
modelb.fit(X_train,y_train)
print("Score of the train set",modelb.score(X_train,y_train))
print("Score of the test set",modelb.score(X_test,y_test))
# With outliers & not scaled
X_train, X_test, y_train, y_test = train_test_split(Xb,yb, test_size=0.3, random_state=42)
modelb = LinearRegression(normalize=False)
modelb.fit(X_train,y_train)
print("Score of the train set",modelb.score(X_train,y_train))
print("Score of the test set",modelb.score(X_test,y_test))
# With outlier but scaling
X_alls = StandardScaler().fit_transform(Xb)
X_train, X_test, y_train, y_test = train_test_split(X_alls,yb, test_size=0.3, random_state=42)

modelb = LinearRegression(normalize=False)
modelb.fit(X_train,y_train)
print("Score of the train set",modelb.score(X_train,y_train))
print("Score of the test set",modelb.score(X_test,y_test))<jupyter_output>Score of the train set 0.7434997532004697
Score of the test set 0.7112260057484923
{'fraction_non_alphanumeric': 0.09302116971654108, 'fraction_numerical': 0.02377108001435235, 'mean_word_length': 6.856941508104299, 'pattern_counts': {'":': 0, '<': 55, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '455506', 'license': 'MIT', 'n_tokens_mistral': 4175, 'n_tokens_neox': 3933, 'n_words': 1107, 'path': 'linear_regression.ipynb', 'repo_name': 'ola-sumbo/GADS2020-Roadmap', 'size': '28784'}