<jupyter_start><jupyter_text>Linear Models in Classification$$ \hat{\mathbf{y}} = f(\sum_{i=1}^{d}x_{i}w_{i} + b) = f(\mathbf{x^{T}w} + b)$$  Where:$d$ = number of features / dimensions$\mathbf{x}$ = vector containing input features$\mathbf{w}$ = vector of weights (or coefficients)$b$ = bias$f$ is some thresholding function Typically:$    f(x)= \begin{cases}    1,& \text{if } x > 0\\    0,              & x \leq 0\end{cases}$<jupyter_code>X, y = utils.make_classification()
plt.figure(figsize=(8, 6))
plt.scatter(X[:, 0], X[:, 1], c=y)
lr = LogisticRegression()
lr.fit(X, y)
plt.figure(figsize=(8, 6))
utils.draw_decision_boundary(lr, X, y)<jupyter_output><empty_output><jupyter_text>Exercise: Try fitting the logistic regression model on the following dataset and plot the decision boundary<jupyter_code>X, y = utils.make_moons(noise=0.01)
plt.figure(figsize=(8, 6))
plt.scatter(X[:, 0], X[:, 1], c=y)
# enter code here<jupyter_output><empty_output><jupyter_text>Question: What went wrong?<jupyter_code>from sklearn.svm import SVC
svc = SVC()
svc.fit(X, y)
utils.draw_decision_boundary(svc, X, y)<jupyter_output><empty_output><jupyter_text>Exercise: Try to fit a SVC to the following dataset and plot the decision boundary<jupyter_code>X, y = utils.make_circles(factor=0.1, noise=0.1)
plt.figure(figsize=(8, 6))
plt.scatter(X[:, 0], X[:, 1], c=y)
# enter code here<jupyter_output><empty_output><jupyter_text>Linear Models in Regression $$ \hat{\mathbf{y}} = \sum_{i=1}^{d}\beta_{i}x_{i} + \epsilon = \mathbf{x^T\beta} + \epsilon$$  Where:$d$ = number of features / dimensions$\mathbf{x}$ = vector containing input features$\mathbf{\beta}$ = vector of coefficients$\epsilon$ = error, residue or nouse<jupyter_code>x = np.arange(100)
y = np.linspace(0, 1, 100) + np.random.rand(100,) * 0.1
plt.figure(figsize=(8, 6))
plt.scatter(x, y)
plt.xlabel('$x$')
plt.ylabel('$y$')
lr = LinearRegression()
lr.fit(x.reshape(-1, 1), y)
y_hat = lr.predict(x.reshape(-1, 1))
plt.figure(figsize=(8, 6))
plt.scatter(x, y, label='original')
plt.plot(x, y_hat, 'g', label='predicted')
plt.legend()<jupyter_output><empty_output><jupyter_text>Examining the coefficients of the trained model<jupyter_code>lr.coef_
lr.intercept_
y_hat_unfit = x * lr.coef_ + lr.intercept_
plt.figure(figsize=(8, 6))
plt.plot(x, y_hat, 'go', x, y_hat_unfit, 'r-')<jupyter_output><empty_output><jupyter_text>Exercise: Fit a linear model to the following dataset and find its slope and intercept.<jupyter_code>X, y = utils.make_regression_exercise()
plt.figure(figsize=(8, 6))
plt.scatter(X, y)
plt.xticks([])
plt.yticks([])
# enter code here
# Examine the source code of `utils.make_regression_exercise` to check answer<jupyter_output><empty_output><jupyter_text>Sparsity in Linear Models and Data Compression  All regression is approximation, solving$$ Ax = b $$ Where:$$ A \in \mathbb{R}^{m \times n} $$ When$m > n$, $A$ is a tall matrix -> overdetermined system of equations$m  underdetermined system of equations An overdetermined system has no solution -> solve an approximation For example, find a solution that produces the least MSE, i.e.$$ \underset{x}{min}\|Ax - b\|^2 $$ An underdetermined system has infinitely many solutions -> impose a constraint on solutions For example, find the _sparsest_ solution  Example: The Simplest Impossible Problem Which two numbers have the mean 3? Arithmetic mean as matrix multiplication:$ A = \begin{bmatrix}0.5 & 0.5 \\0 & 0\end{bmatrix}$$b = \begin{bmatrix}3\\0\end{bmatrix} $$x = \begin{bmatrix}x_{1}\\x_{2}\end{bmatrix}$ Then solve$Ax = b$<jupyter_code>A = np.array([[0.5, 0.5], [0, 0]])
b = np.array([[3], [0]])
lr.fit(A, b)  # Linear Regression
print(lr.coef_)
lasso = Lasso(alpha=0.0001)
lasso.fit(A, b)
print(lasso.coef_)<jupyter_output><empty_output><jupyter_text>Example: DTMF - Linear Combination of Two Sinusoids![](dtmf.jpg)<jupyter_code>Fs = 32768
duration = 0.25
t = np.linspace(0, duration, duration * Fs)
f1, f2 = 697, 1336
y1 = np.sin(2 * np.pi * f1 * t);
y2 = np.sin(2 * np.pi * f2 * t);
y = (y1 + y2) / 2
plt.figure(figsize=(8, 6))
plt.plot(t, y)
from IPython.display import Audio
Audio(y, rate=44100)<jupyter_output><empty_output><jupyter_text>Repeat the signal ten times<jupyter_code>t = np.linspace(0, duration * 10, duration * 10 * Fs)
f1, f2 = 697, 1336
y1 = np.sin(2 * np.pi * f1 * t);
y2 = np.sin(2 * np.pi * f2 * t);
y = (y1 + y2) / 2
Audio(y, rate=44100)<jupyter_output><empty_output><jupyter_text>Recreate the original signal for simplicity<jupyter_code>t = np.linspace(0, duration , duration * Fs)
f1, f2 = 697, 1336
y1 = np.sin(2 * np.pi * f1 * t);
y2 = np.sin(2 * np.pi * f2 * t);
y = (y1 + y2) / 2<jupyter_output><empty_output><jupyter_text>Randomly sampling the signal<jupyter_code>N = y.shape[0]  # length of the signal
M = 800  # number of samples
plt.figure(figsize=(10, 8))
plt.subplot(211), plt.plot(t, y)
plt.xlim(0, 0.125)
plt.title('Original Signal')
# Randomly sampling the test signal
k = np.random.randint(0, N, (M,))
k = np.sort(k) # making sure the random samples are monotonic
b = y[k]
plt.subplot(212), plt.plot(t, y, 'b', t[k],b,'r.')
plt.xlim(0, 0.125)
plt.title('Original Signal with Random Samples')<jupyter_output><empty_output><jupyter_text>Discrete Cosine Coefficients as the data that "predict" the signal Or, which signal, when operated on by a DCT, will produce the sampled signal?<jupyter_code>D = dct(np.eye(N), axis=0)
A = D[k,:]
lasso = Lasso(alpha=0.001)
lasso.fit(A, b)
print(lasso.coef_)
print('Sparsity {} %'.format((lasso.coef_ != 0).sum() / lasso.coef_.shape[0] * 100))
recons = idct(lasso.coef_)
plt.figure(figsize=(10, 8))
plt.subplot(211), plt.plot(recons)
plt.title('Reconstructed signal')
plt.subplot(212), plt.plot(np.linspace(0, Fs / 2, N), lasso.coef_), plt.xlim(0, 2500)
plt.title('Sparse Coefficients')<jupyter_output><empty_output><jupyter_text>Thresholding the coefficients:<jupyter_code>coefs = lasso.coef_.copy()
coefs[np.abs(coefs) <= 0.1] = 0
recons_th = idct(coefs)

plt.figure(figsize=(10, 8))
plt.subplot(211), plt.plot(recons_th)
plt.title('Reconstructed signal')
plt.subplot(212), plt.plot(np.linspace(0, Fs /2, N), coefs), plt.xlim(0, 2500)
plt.title('Sparse Coefficients (Thresholded)')
Audio(np.tile(recons_th, 10),  rate=44100)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1691963572455664, 'fraction_numerical': 0.037386163923949514, 'mean_word_length': 5.673773987206823, 'pattern_counts': {'":': 0, '<': 58, '<?xml version=': 0, '>': 62, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '217864', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2666, 'n_tokens_neox': 2454, 'n_words': 587, 'path': '04_linear_models.ipynb', 'repo_name': 'jaidevd/talentsprint-workshop', 'size': '6259'}