<jupyter_start><jupyter_text>Linear Regression Agenda:- Introduction of Regression Problems- Introducing `Advertising` Dataset- Train-Test Dataset split - Discussion of Liner Regression in the Simple Setting    - Model    - Hypothesis Function    - Cost Function    - Solution - Least Squares Approach    - MultiLinear Regression- Residual Plots  Dataset- We will be using the `advertising` dataset from ISLR- This data contains advertising budget in (Thousand $\$$) and product sales (Millions $\$$)- Let's take a look at the dataset<jupyter_code>import pandas as pd # for manipulating data
import matplotlib.pyplot as plt # for visualizations
import seaborn as sns # for visualizations
import requests # for loading data from an online resource
from io import StringIO # for reading inputs

orig_url = "https://drive.google.com/file/d/1XdOaGHuGWgEMVMEiJ0dbwdPh6mH6PQZi/view?usp=sharing"

file_id = orig_url.split('/')[-2]

dwn_url='https://drive.google.com/uc?export=download&id=' + file_id

url = requests.get(dwn_url).text

csv_raw = StringIO(url)

advertising = pd.read_csv(csv_raw, index_col= 0)
advertising[["TV", "Sales"]].head()
advertising.shape<jupyter_output><empty_output><jupyter_text>__Goal__Goal for this project is to be able to predict `sales` given the advertising budgets of the company in some other city. __Note__- This is a `supervised learning` problem: ( We are given `right answers` )- More particularly this is a Regression Problem (Predicting a real-valued output) __Training vs Test Datasets__- We want `machine` to learn from our training data.- Also don't forget the ultimate goal: Accurately predicting sales for an unseen data(New city) when we apply the algorithm to this case.- But `machine`'s success on training data might be misleading! (Do you see why?)  - Can you think of a solution?<jupyter_code>## train test split

from sklearn.model_selection import train_test_split

X_train, X_test = train_test_split(advertising, test_size = 0.2) # splits the data into two parts with 1:4 ratio
print(X_train.shape)
X_train[["TV", "Sales"]].head()
print(X_test.shape)
X_test[["TV", "Sales"]].head()
class display(object):
    """Display HTML representation of multiple objects"""
    template = """<div style="float: left; padding: 10px;">
    <p style='font-family:"Courier New", Courier, monospace'>{0}</p>{1}
    </div>"""
    def __init__(self, *args):
        self.args = args
        
    def _repr_html_(self):
        return '\n'.join(self.template.format(a, eval(a)._repr_html_())
                         for a in self.args)
    
    def __repr__(self):
        return '\n\n'.join(a + '\n' + repr(eval(a))
                           for a in self.args)
print("Train Size", X_train.shape, "Test Size: ", X_test.shape, sep = "\n")
display('X_train[["TV", "Sales"]].head()', 'X_test[["TV", "Sales"]].head()')<jupyter_output>Train Size
(160, 4)
Test Size: 
(40, 4)<jupyter_text>Linear Regression: Simple Setting [Image is inspired from](https://www.youtube.com/watch?v=kHwlB_j7Hkc&ab_channel=ArtificialIntelligence-AllinOne) __Question__- How do we represent $f$?- In Linear Regression Problems:  $$ f(x) = \beta_{0} + \beta_{1} x $$<jupyter_code>sns.lmplot(x = 'TV', y = 'Sales', data = advertising, ci = 0)
plt.draw()<jupyter_output><empty_output><jupyter_text>Goal: Minimize the errors! - What errors?- How? __Toy case__ Note that $$ e_i = y_{i} - (\beta_{0} + \beta_{1}x_{i})$$ __Errors__ __What is the total sum of errors?__ $$ e_{1} + e_{2} + \cdots + e_{N}$$- Can you see the problem with this? __Better Way to Measure Errors__ __Residual Sum of Squares__$$ RSS = e_{1}^{2} + e_{2}^{2} + \cdots + e_{N}^{2} $$__Still room for improvement__ (Mean Squared Error)$$ MSE = \frac{1}{N}\left(e_{1}^{2} + e_{2}^{2} + \cdots + e_{N}^{2} \right) $$ __Or in short__$$ MSE = \frac{1}{N}\left( \sum_{i =1}^{N} e_{i}^{2} \right) $$  __Note that `MSE` is a function of $\beta_0, \beta_{1}$:__$$ \mbox{MSE} = \frac{1}{N} (y_1 -\beta_{0} -\beta_{1}x_{1} )^{2} + (y_2 -\beta_{0} -\beta_{1}x_{2} )^{2} + \cdots + (y_N -\beta_{0} -\beta_{1}x_{N} )^{2}$$ __Hypothesis Function__(Fix $\beta_0, \beta_{1}$ function of $x$)$$ y = f(x) = \beta_{0} + \beta_{1} x $$__Cost Function__(This is a function of $\beta_0, \beta_{1}$)$$ \mbox{MSE} = \frac{1}{N} (y_1 -\beta_{0} -\beta_{1}x_{1} )^{2} + (y_2 -\beta_{0} -\beta_{1}x_{2} )^{2} + \cdots + (y_N -\beta_{0} -\beta_{1}x_{N} )^{2}$$ __How :)__ (Least Squares Approach)  [Image Source: <NAME>](https://www.youtube.com/watch?v=0kns1gXLYg4&ab_channel=ArtificialIntelligence-AllinOne)<jupyter_code>import matplotlib.image as mpimg 

f, axes = plt.subplots(nrows= 1, ncols = 3)
for i in range(3):
    axes[i].axis('off')
    if i==0:
        img = mpimg.imread(f'images/simple{i}.jpg')
    else:
        img = mpimg.imread(f'images/simple{i}.png')
    axes[i].imshow(img)
plt.gcf().set_size_inches(20, 20)<jupyter_output><empty_output><jupyter_text>Mathematically we will write:$$\boldsymbol{y}  = X\cdot \boldsymbol{\beta} $$ __Seriously How!__ - Direct Solution!$$ \boldsymbol{\beta} = (X^{T}X)^{-1}X^{T}\boldsymbol{y}$$<jupyter_code>def least_squares(X,y):
    x_bar = X.mean()
    y_bar = y.mean()
    b1_num = (X - x_bar).dot(y - y_bar)
    b1_denom = ((X - x_bar)**2).sum()
    b1 = b1_num /b1_denom
    b0 = y_bar - b1*x_bar
    MSE = ((y - (b0 + b1*X))**2).mean()
    return  b0,b1, MSE

least_squares(X_train.TV, X_train.Sales)
## we could use numpy to solve this directly

import numpy as np
X = np.array(X_train.TV)

ones = np.ones(shape = X.shape)

X = np.c_[ones, X]
X_Trans = X.T

np.linalg.inv( X_Trans.dot(X)).dot(X_Trans).dot(X_train.Sales)<jupyter_output><empty_output><jupyter_text>__Or Gradient Descent and its Variations__ __Or Using Sklearn__<jupyter_code>## Four Step workflow with sklearn
## 1. import the tool
from sklearn.linear_model import LinearRegression

## 2. instantiate the tool you would like to use.
lr = LinearRegression()

## 3. fit the model
lr.fit(X_train[["TV"]], X_train.Sales)

## 4. check-out the results or make predictions
print(lr.intercept_, lr.coef_)<jupyter_output>7.249357542276544 [0.0480755]<jupyter_text>Part-II: More than One Variables - When we have more than one variable our linear model will look like as:$$ Y = \beta_{0} + \beta_{1} X_{1} + \beta_{2}X_{2} + \cdots \beta_{p}X_{p} + \epsilon$$ __Notation:__- $X_{1}, \cdots, X_{p}$ :  Columns of the dataset (or features or predictors or independent variables)- $Y$ : target column in the dataset (or target variable or dependent variable)- $\epsilon$ : Irreducible error.__Goal:__ Given a dataset $X$ we would like to find coefficients $\beta_{1}, \cdots, \beta_{p}$ such that the sum of squared errors are minimum (Least Squares Approach!) - Cost function (objective function) become:\begin{align}RSS &= \sum\limits_{i=1}^{n} (y_{i} - \hat{y}_{i})^{2} \\&= \sum\limits_{i=1}^{n} (y_{i} - \beta_{0} - \beta_{1} x_{i1}  - \beta_{2}x_{i2} - \cdots  - \beta_{p}x_{ip})^{2}\end{align}<jupyter_code>## This time we will use statsmodels api

import statsmodels.api as sm

X = X_train[['TV', 'Radio', 'Newspaper']]
y = X_train.Sales
X = sm.add_constant(X)
mod = sm.OLS(y, X, hasconst= True)
res = mod.fit()
print(res.summary())<jupyter_output>OLS Regression Results                            
==============================================================================
Dep. Variable:                  Sales   R-squared:                       0.909
Model:                            OLS   Adj. R-squared:                  0.907
Method:                 Least Squares   F-statistic:                     519.9
Date:                Tue, 15 Sep 2020   Prob (F-statistic):           5.75e-81
Time:                        16:23:32   Log-Likelihood:                -298.35
No. Observations:                 160   AIC:                             604.7
Df Residuals:                     156   BIC:                             617.0
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|[...]<jupyter_text>__Is this a good model?__<jupyter_code>Xtest = X_test[['TV', 'Radio', 'Newspaper']]
ytest = X_test.Sales
Xtest = sm.add_constant(Xtest)

res.predict(Xtest);
## To diagnose a linear model 
## it's always a good idea to check residual plots

residuals = res.resid

y_pred = res.predict(X)
plt.scatter(y_pred, residuals)
plt.axhline(y=0, color = 'red', label = '0')

plt.xlabel('predicted values')

plt.ylabel('residuals')

plt.tight_layout()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.16643598615916955, 'fraction_numerical': 0.02295271049596309, 'mean_word_length': 3.478822314049587, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 36, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '158807', 'license': 'MIT', 'n_tokens_mistral': 3178, 'n_tokens_neox': 2977, 'n_words': 815, 'path': 'lectures/week-3/Lecture/Linear Regression.ipynb', 'repo_name': 'dbbabcock/UMBC_DATA602', 'size': '8673'}