<jupyter_start><jupyter_text>Linear Regression```{note}Linear Regression = Linear Model + Mean Square LossWe can use either gradient descent or analytic solution to solve linear regression.Linear Regression has nice geometric and probabilistic Interpretations.```  ModelSuppose $x \in \mathbb{R}^{d}$, $y \in \mathbb{R}$. Linear model is:$$h(x) = w^{T}x + b$$For simplicity, let:$$x := [x,1] \in \mathbb{R}^{d + 1}$$$$\theta := [w, b] \in \mathbb{R}^{d + 1}$$Then linear model could be write as:$$h(x) = \theta^{T}x$$![jupyter](images/linear_regression1.svg)  LossLoss Function is mean square loss:$$J(\theta) = \frac{1}{2}\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})^{2}$$  Update RuleGradient Descent:$$\theta \to \theta - \alpha\nabla{J(\theta)}$$Gradient of Linear Regression:$$\begin{equation}\begin{split} \frac{\partial }{\partial \theta_{j}}J(\theta) &=  \frac{\partial }{\partial \theta}_{j}\frac{1}{2}\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})^2  \\ &=\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})\cdot\frac{\partial }{\partial \theta_{j}}(h(x^{(i)}) - y^{(i)})\\ & =\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})\cdot{x_{j}}^{(i)}\end{split}\end{equation}$$Combine all dimensions:$$\theta \to \theta - \alpha\sum_{i=1}^{n}(h(x^{(i)}) - y^{(i)})\cdot{x^{(i)}} $$Write in matrix form:$$\theta \to \theta - \alpha{X^{T}}(X\theta-y) $$where $X \in \mathbb{R}^{n\times{d}}, y \in \mathbb{R}^{n}$.  Analytic SolutionFrom above, we have:$$\nabla{J(\theta)} = X^{T}X\theta - X^{T}y$$If $X^{T}X$ is invertible:$$\theta = (X^{T}X)^{-1}X^{T}y$$Else the equation also have solution:$$X^{T}X\theta=0 \Rightarrow \theta^{T}X^{T}X\theta = (X\theta)^{T}X\theta=0 \Rightarrow X\theta = 0$$$$\mbox{null}(X^{T}X) = \mbox{null}(X) \Rightarrow \mbox{range}(X^{T}X) = \mbox{range}(X^{T})$$  Examples<jupyter_code>from sklearn.datasets import load_boston

X, y = load_boston(return_X_y=True)
X.shape, y.shape
from sklearn.linear_model import LinearRegression

reg = LinearRegression()
reg.fit(X, y)
from sklearn.metrics import mean_squared_error
mean_squared_error(y, reg.predict(X))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.25918762088974856, 'fraction_numerical': 0.00918762088974855, 'mean_word_length': 5.8737541528239205, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '551114', 'license': 'MIT', 'n_tokens_mistral': 824, 'n_tokens_neox': 762, 'n_words': 157, 'path': 'machine-learning-book/a1.linear regression.ipynb', 'repo_name': 'newfacade/jupyters', 'size': '2068'}