<jupyter_start><jupyter_text>Linear Regression> This tutorial describes linear regression technique and demonstrates how it works via an example of fitting a curve using linear regression.  Regression Regression is an approach to modeling the relationship between a real-valued target $y$ (dependent variable) and data points $\mathbf{x}$ (independent variables). In other words, linear regression describes the relationship between input and output and predicts the output based on the input data.>Note: **x** is a vector of features, i.e. $\mathbf{x} = $Examples of linear regression include, predicting the weight from gender, age, and height, or predicting the stock price today from the prices of yesterday.We wish to learn $f:X\to Y$, where $Y$ is a real number, given $\{,\cdots, \}$.**Approach**1- Choose some parametraized form for $P(Y|X;\theta)$2- Derive learning algorithms as Maximum Likelihood Estimates (MLE), or Maximum a Posteriori (MAP) estimate for $\theta$  Linear Regression Model Let's assume that we are going to predict the price of houses (in dollars), $y$, based on their area (in square feet), $x$. Therefore, training set = $\{,\cdots,\}$, where:- $m$: number of trainig examples- $x$: input variable- $y$: output variable or target label- $(x^i,y^i)$: is the $i$th training example<jupyter_code>#hide
import numpy as np
import pandas as pd
import altair as alt

# Generate some random data
rng = np.random.RandomState(1)
x = rng.rand(40) ** 2
y = 10 - 1.0 / (x + 0.1) + rng.randn(40)
source = pd.DataFrame({"square feet": x, "price": y})

# Define the degree of the polynomial fits
degree_list = [1, 3, 5, 9]

base = alt.Chart(source).mark_circle(color="black").encode(
        alt.X("square feet", axis=alt.Axis(labels=False)), 
        alt.Y("price", axis=alt.Axis(labels=False))
)

polynomial_fit = [
    base.transform_regression(
        "square feet", "price", method="poly", order=order, as_=["square feet", str(order)]
    )
    .mark_line()
    .transform_fold([str(order)], as_=["degree", "price"])
    .encode(alt.Color("degree:N"))
    for order in degree_list
]

# chart = alt.layer(base, *polynomial_fit)
# chart.save("lr_house_ex.png", scale_factor=2)
# chart = alt.layer(base,polynomial_fit[0])
# chart.save("lr_house_line_ex.png", scale_factor=2)<jupyter_output><empty_output><jupyter_text>We may think of a line as the regression model to predict the prices of houses from their area.Someone might consider a quadratic function or even high order polynomial functions to do the prediction.  Polynomial Regression ModelIf we represent the relationship between the independent variable $x$ and dependent variable $y$ as an $n$th degree polynomial, then the regression model is called **polynomial regression**. Therefore, we have:$$y_i=w_0+w_1x_i+w_2x_i^2+\cdots+w_nx_i^n+\epsilon_i$$We treat each $x_i^p$ for $p=1,\cdots,n$ as a different feature. i.e. $feature\ 1=x, feature\ 2=x^2,\cdots, feature\ n=x^n$.  General ExpansionIn general, if we have a dataset of $m$ instances with $k$ features, $\mathbf{x}=$ and a real valued target $y$, then the linear regression model takes the form:$$y_i=w_0+w_1h_1(x_1)+w_2h_2(x_2)+\cdots+w_kh_k(x_k)+\epsilon_i= w_0 +\sum_{j=1}^{k}w_jh_j(x_i)+\epsilon_i$$Considering our house price prediction, instead of using only area of the house, we can add more features such as number of bathrooms, number of bedrooms, age of the house, etc.  The Regression ProblemThe general approach is as follows:- Instances: $$- Learn: mapping from $\mathbf{x}$ to $y\equiv f(\mathbf{x})$- Given the basis functions $h_1, h_2,\cdots,h_k$ where $h_j(\mathbf{x})\in \mathbb{R}$- Find coeffcients $\mathbf{x}=[w_0,w_1,\cdots,w_k]$, where $f(\mathbf{x})\thickapprox \hat{f}(\mathbf{x})=w_0+\sum_{j}w_jh_j(\mathbf{x})$- In order to find coefficients, we need to minimize the **mean residual square error**:$$J(\mathbf{w}) = \frac{1}{m}\sum_{i=1}^{m}\left( f(\mathbf{x}_i) - \left(w_0 +\sum_{j=1}^{k}w_jh_j(\mathbf{x})\right)\right)^2$$$$\mathbf{w}^* = \underset{\mathbf{w}}{\arg\min}\ J(\mathbf{w})$$>Note: The reason it's called *linear* regression is that the model is a linear combinations of the input features.  Example: Fit a Line to Two Diminsional Data We would like to find a **line** that **fits** our training examples the **best**. Since we are looling for a line to fit our training set, we need to find a function $\hat y = f(x)$ that estimates $y^i$ for all $1\leq i \leq m$. Thus, we have:$\hat y = f(x) = w_0 + w_1x$$w_0$ and $w_1$ are the paramaters that we need to find.  Cost Function (Error Function)We wish to estimate the actual $y$, i.e. we want to minimize the error as much as possible. The error is the difference between the actual $y^i$ and our estimated/predicted $\hat{y}^i$ for all $1\leq i \leq m$. The error/cost function is: $$J(w_0,w_1) = \frac{1}{m}\sum_{i=1}^{m}\left( y^i - (w_0 + w_1 x^i)\right)^2$$ Precisely, we need to minimize the **mean residual squared error**:$$\text{minimize } J(w_0,w_1) = \underset{\mathbf{w0,w_1}}{\arg\min} \frac{1}{m}\sum_{i=1}^{m}\left( \hat{y}^i - (w_0 + w_1 x^i)\right)^2$$  Fitting the Linear Regression Model using Gradient Descent AlgorithmIn order to mininze the cost function, we need to take the gradient (i.e. derivative) of the function with resptect to our parameters $w_0$ and $w_1$, set it zero and solve for $w_0$ and $w_1$.  $$\frac{\partial{J}}{\partial{w_0}} = \frac{-2}{m} \sum_{i=1}^{m}\left(\hat{y}^i - (w_0 + w_1 x^i)\right)$$$$\frac{\partial{J}}{\partial{w_1}} = \frac{-2}{m} \sum_{i=1}^{m}x^i\left(\hat{y}^i - (w_0 + w_1 x^i)\right)$$  The gradient descent algorithm:initialize $w_0^{(0)} = w_1^{(0)} = 0$, for $t=0$for $t=1$ to *num_iterations*      &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$w_0^{(t+1)} = w_0^{(t)} - \eta \frac{-2}{m} \sum_{i=1}^{m}\left(\hat{y}^i - (w_0 + w_1 x^i)\right) $&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$w_1^{(t+1)} = w_1^{(t)} - \eta\frac{-2}{m} \sum_{i=1}^{m}x^i\left(\hat{y}^i - (w_0 + w_1 x^i)\right)$&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$t = t+1$<jupyter_code># export
import numpy as np
import pandas as pd
import altair as alt
def generate_data():
    """It generates dummy data."""
    noise = np.random.randn(100,1)
    X = 2 * np.random.rand(100,1)
    y = 5 + 3 * X + noise
    return X,y
# export
def make_point_plot(x,y):
    """It plots the point chart of the data"""
    
    data_points = pd.DataFrame({'x': x.flatten(), 'y': y.flatten()})
    chart = alt.Chart(data_points).mark_point(size=50, color='red',filled=True).encode(
        x="x",
        y="y"
    )
    return chart
    
# export
def make_line_plot(x,y):
    """It plots the line chart of the data"""
    
    data = pd.DataFrame({'x': x.flatten(), 'y': y.flatten()})
    line = alt.Chart(data).mark_line(size=3).encode(
        x="x",
        y="y"
    )
    return line
#export
def gradient_descent(data,w_0_t,w_1_t,learning_rate,num_iterations):
    """Gradient descent implementation, which gets `data`, starting `w_0` and `w_1`, `learning_rate` 
    and the number of iterations `num_iterations`"""
    
    w_0 = 0
    w_1 = 0
    (X,y) = data
    N = len(X)
    w_0_t = 0
    w_1_t = 0
    for t in range(0,num_iterations):
        w_0_deriv = np.zeros((N,N))
        w_1_deriv = np.zeros((N,N))
        w_0_deriv = -2 * (y - (w_0_t + w_1_t * X))
        w_1_deriv = -2 * np.dot(X.T, (y - (w_0_t + w_1_t * X)))
        w0_sum = np.sum(w_0_deriv,axis=0)
        w1_sum = np.sum(w_1_deriv,axis=0)
        w_0 = w_0 - learning_rate * (w0_sum / N)
        w_1 = w_1 - learning_rate * (w1_sum / N)
        w_0_t = w_0
        w_1_t = w_1
    return w_0, w_1<jupyter_output><empty_output><jupyter_text>Run an ExampleIn this example, we generate some linear-looking data and then find the line that fits the data. The function that we use to generate the data is $y=5+3x+\text{Gaussian noise}$. Our goal is to find $\theta=[w_0,w_1]$ where $w_0=5$ and $w_1=3$ or close enough, as we have noise and it makes it impossible to recover the exact paratmeters of the function. We use the `generate_data` function to generate the training data.<jupyter_code>X,y = generate_data()<jupyter_output><empty_output><jupyter_text>The chart below shows the generated dataset.<jupyter_code>origin_chart = make_point_plot(X,y)

#show the chart
origin_chart<jupyter_output><empty_output><jupyter_text>We set the initial parameters to 0:  $w_0^{(0)} = w_1^{(0)} = 0$, define the number of iterations of the graditent descent algorithm as well as the learning rate. Then, we run the `gradient_descent` function. The outputs of this function are the estimated parameters $w_0$ and $w_1$.<jupyter_code>initial_w_0 = 0
initial_w_1 = 0
num_iterations = 1000
learning_rate = 0.01

# X,y = generate_data()
data = (X,y)
w_0,w_1 = gradient_descent(data,initial_w_0,initial_w_1,learning_rate,num_iterations)

print("w_0 = {}, w_1 = {}".format(w_0,w_1))<jupyter_output>w_0 = [4.60926576], w_1 = [3.24869531]<jupyter_text>The estimated parameters are $w_0 = 4.966$ and $w_1 = 3.001$ that are close enough to $w_0=5$ and $w_1=3$. Let's plot the line.<jupyter_code>y_predicted = w_0 + w_1 * X
line = make_line_plot(X,y_predicted)

#show the charts
origin_chart + line
# hide

num_iterations = 1000
l_rates = [0.001,0.1,0.002]
learning_rate = 0.001
all_params = []

# X,y = generate_data()
data = (X,y)
for i in range(0,len(l_rates)):
    initial_w_0 = 0
    initial_w_1 = 0
    learning_rate = l_rates[i]
    w_0,w_1 = gradient_descent(data,initial_w_0,initial_w_1,learning_rate,num_iterations)
    all_params.append((w_0,w_1))
    print("learning_rate: {:.4f} => w_0 = {}, w_1 = {}".format(learning_rate,w_0,w_1))<jupyter_output>learning_rate: 0.0010 => w_0 = [3.7161119], w_1 = [3.92225456]
learning_rate: 0.1000 => w_0 = [4.74240926], w_1 = [3.13681352]
learning_rate: 0.0020 => w_0 = [3.95081957], w_1 = [3.80117934]
{'fraction_non_alphanumeric': 0.15313484401991667, 'fraction_numerical': 0.03454933441723402, 'mean_word_length': 4.4617092119866815, 'pattern_counts': {'":': 2, '<': 23, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '315233', 'license': 'Apache-2.0', 'n_tokens_mistral': 3724, 'n_tokens_neox': 3550, 'n_words': 1065, 'path': '03_linear_regression.ipynb', 'repo_name': 'mjc3bb/ml_tutorial', 'size': '9841'}