<jupyter_start><jupyter_text>Homework 3**For exercises in the week 11-16.12.19****Points: 7 + 1b**Please solve the problems at home and bring to class a [declaration form](http://ii.uni.wroc.pl/~jmi/Dydaktyka/misc/kupony-klasyczne.pdf) to indicate which problems you are willing to present on the blackboard.$\def\R{{\mathbb R}} \def\i{^{(i)}} \def\sjt{\mathrm{s.t. }\ }$  Problem 1 [2p]Let $X\in \R^{D\times N}$ be a data matrix contianing $N$ $D$-dimensional points. Let $Y\in\R^{1\times N}$ be the targets.We have seen that the least squares problem$$\min_{\Theta} \frac{1}{2}(\Theta^T X - Y)(\Theta^T X - Y)^T$$has a closed form solution$$\Theta^T{}^* = Y X^T(X X^T)^{-1}$$Where $X^+ = X^T(X X^T)^{-1}$ is the right [Moore-Penrose pseudoinverse](https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse) of $X$:$$\begin{split}\Theta^T X &\approx Y \\\Theta^T X X^+ &\approx Y X^{+} \\\Theta^T &= Y X^{+}\end{split}$$The pseudoinverse also has another form (called a left inverse):$$X^+ = (X^T X)^{-1}X^T$$ P1.1 [0.5p]Say under which conditions the left and right pseudoinverses exist (when $X$ is a rectangular matrix only one index exists). Give examples of machine learning problems that could be solved using each inverse. Ans:If A is m-by-n and n ≤ m, then A has a left inverse. \If m ≤ n, then it has a right inverse. \right -> lin reg P1.2 [1p]Derive the left inverse by solving the regularized least squares problem$$\min_\Theta \sum_i(\Theta^T x\i - y\i)^2 + \lambda\Theta^T\Theta$$with arificially introduced variables $\epsilon\i$ and constraints $\epsilon\i = \Theta^T x\i - y\i$, then see what happens when $\lambda\rightarrow 0$. P1.3 [0.5p]Show that the above dual formulation allows using Kernel functions with linear regression. Express the optimal solution using a weighed avegage of data samples. How many "support vectors" there are?NB: some authors call the kernelized linear regression the "Least-Squares SVM".<jupyter_code>import numpy as np
from numpy.linalg import pinv, inv

X = np.random.random((500, 3)) * 1000
left = lambda x: inv(x.T @ x) @ x.T
right = lambda x: x.T @ inv(x @ x.T)

(left(X) @ X).astype(int), (X @ right(X)).astype(int), X.astype(int)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15787099684258007, 'fraction_numerical': 0.02029769959404601, 'mean_word_length': 4.383495145631068, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 6, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '205988', 'license': 'MIT', 'n_tokens_mistral': 791, 'n_tokens_neox': 741, 'n_words': 272, 'path': 'homework3/Homework3.ipynb', 'repo_name': 'iCarrrot/ML', 'size': '2217'}