<jupyter_start><jupyter_text>**Training samples**: a cubic polynomial with noise**Goal:** Learn a degree 'd' Legendre polynomial on training samples. First we do some imports<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

import ipywidgets
from ipywidgets import interact<jupyter_output><empty_output><jupyter_text>and set some seeds<jupyter_code>np.random.seed(1932)<jupyter_output><empty_output><jupyter_text>Now, let's define the function space<jupyter_code>H = np.polynomial.legendre.legvander # Legendre polynomial basis
x = 2
H(x, 5)<jupyter_output><empty_output><jupyter_text>Let's define the ground truth to be cubic<jupyter_code>def cubic_gt(x):
  beta = np.array([0, -1, 0, 1])
  return H(x, 3).dot(beta)<jupyter_output><empty_output><jupyter_text>and sample some points from this ground truth<jupyter_code>n = 20
sigma = 0.3 # stdev of noise

## Compute ground-truth polynomial + noise
x = np.linspace(-1, 1, n)
y = cubic_gt(x)

y += (np.random.uniform(size=n) <= 0.2)*sigma # random L0 noise
y -= (np.random.uniform(size=n) <= 0.1)*2*sigma # random L0 noise<jupyter_output><empty_output><jupyter_text>generate a denser grid to plot<jupyter_code>xdense = np.linspace(-1, 1, 100) # points to plot
xdense = np.sort(np.concatenate((x, xdense))) # add in the sample points explicitly
ygt = cubic_gt(xdense)<jupyter_output><empty_output><jupyter_text>Note that the output of minimizing $$\min_\beta ||X \beta - y||^2_2$$is equivalent to $\hat{\beta} = X^\dagger y$. So, the fit and prediction from our model<jupyter_code>class MyClassifier:
    def __init__(self, d):
        self.d = d

    def fit(self, x, y):
        self.beta_ = np.linalg.pinv(H(x, self.d)).dot(y)
        
    def predict(self, x):
        return H(x, self.d).dot(self.beta_)
    
    def risk(self, x, y):
        return np.linalg.norm(y - self.predict(x))
def run(d):

  clf = MyClassifier(d)
  clf.fit(x, y)
  
  plt.figure(figsize=(7, 7))
  plt.plot(xdense, ygt, label='ground-truth')
  plt.scatter(x, y, c='r', label='samples')
  plt.plot(xdense, clf.predict(xdense), label='model')
  plt.ylim(-2, 2)
  plt.axis('off')
  plt.legend()
interact(run, d=ipywidgets.IntSlider(min=1, max=20, step=1, value=1))<jupyter_output><empty_output><jupyter_text>Plotting the test risk<jupyter_code>risks = list()
beta_norms = list()
for d in range(1, 100):
  clf = MyClassifier(d)
  clf.fit(x, y)
  beta_norms.append(np.linalg.norm(clf.beta_))
  risks.append(clf.risk(xdense, ygt))
plt.plot(range(1, 11), risks[:10])
plt.axvline(3, linestyle='--')<jupyter_output><empty_output><jupyter_text>The interpolation threshold<jupyter_code>plt.plot(range(1, 100), risks)
plt.axvline(n, linestyle='--')<jupyter_output><empty_output><jupyter_text>**Exercise**: Try changing n  Plotting the norm of the betas<jupyter_code>plt.plot(range(1, 100), beta_norms)
plt.axvline(n, linestyle='--')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15134761575673808, 'fraction_numerical': 0.020041465100207326, 'mean_word_length': 5.447661469933185, 'pattern_counts': {'":': 0, '<': 43, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '447247', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1181, 'n_tokens_neox': 1119, 'n_words': 240, 'path': 'index.ipynb', 'repo_name': 'jasmainak/double-decent-demo', 'size': '2894'}