<jupyter_start><jupyter_text>Overfitting IMajor parts of this lecture are borrowed from the discussion of [Hyperparameters and Model Validation](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html) <NAME>'s excellent online book, the [Python Data Science Handbook](https://jakevdp.github.io/PythonDataScienceHandbook/index.html). In the last lecture, we saw that a reasonably simple decision tree was able to achieve almost 80\% accuracy in modeling from demographic data whether or not a given passenger survived the Titanic. However, we shouldn't take this result at face value. To see why, let's take a look at an example, in the context of regression (i.e. predicting a quantitative variable).<jupyter_code>import numpy as np
from matplotlib import pyplot as plt
np.random.seed(1234)
X = np.random.rand(10)
y = np.log(X) + 0.1*np.random.randn(len(X))
fig, ax = plt.subplots(1)
ax.scatter(X, y)<jupyter_output><empty_output><jupyter_text>Now we're going to do *polynomial regression*. This is the simplest version of what's often called "curve-fitting." In this case, we'll allow the degree of the polynomial to vary. Degree 1 corresponds to linear regression, while higher degrees allow for more complicated curves. Since this is just an illustrative example, it's not necessary for you to understand how this function works at the moment.<jupyter_code>from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

def PolynomialRegression(degree=2, **kwargs):
    return make_pipeline(PolynomialFeatures(degree),
                         LinearRegression(**kwargs))<jupyter_output><empty_output><jupyter_text>The return value of `PolynomialRegression()` is a model, which we can then fit. There is an unfortunate syntactic requirement here -- we need to transform `X`, which is currently a 1d array, into a 2d array in which the second dimension has length 1.<jupyter_code>X[:,None].shape
model1 = PolynomialRegression(1).fit(X[:,None], y)
model2 = PolynomialRegression(10).fit(X[:,None], y)<jupyter_output><empty_output><jupyter_text>That's it for model fitting! Let's check quantitatively how we did:<jupyter_code>model1.score(X[:,None], y), model2.score(X[:,None], y)<jupyter_output><empty_output><jupyter_text>The way that the score is computed depends on the model, but for this class of models the highest possible value of the score is 1. So, `model2` achieves essentially a perfect score, whereas `model1` achieves a noticeably lower score. Let's visualize to see how `model2` achieved such an impressive score.<jupyter_code>xfit = np.linspace(0.01, 1, 1000)[:, None]

ax.plot(xfit, model1.predict(xfit), color = "black")
ax.plot(xfit, model2.predict(xfit), color = "firebrick")

ax.set(ylim = (y.min() - 0.1, y.max() + 1))

fig<jupyter_output><empty_output><jupyter_text>The black line is the simple, degree 1 model, while the wavy red line is more complicated, degree 10 model. Note something interesting: the degree 10 model fits each of the data points exactly! Perfect prediction, no error! Machine learning is easy.    Of course, the situation isn't really that simple. Remember that what we usually want to do in machine learning is not just fit a curve to data, but actually **predict unseen data**. This is the extra wrinkle that we didn't talk about in our first machine learning lecture. To see the problem with `model2`, let's generate some new, unseen data and see how the two models do on the prediction task.<jupyter_code>X_new = np.random.rand(10)
y_new = np.log(X_new) + 0.1*np.random.randn(len(X_new))
model1.score(X_new[:,None], y_new), model2.score(X_new[:,None], y_new)<jupyter_output><empty_output><jupyter_text>Woops! This time, `model1`'s score is similar to what it achieved previously, but `model2`'s score is actually negative-very negative. The useless model that just guesses `0` for each prediction would have yielded a score of `0`, so `model2` is literally worse than useless. We can see this when we visualize: `model2` perfectly fits the original points, but is often badly off on the new ones. In contrast, `model1` is pretty close.<jupyter_code>ax.scatter(X_new, y_new, color = "orange")
fig<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0915640250987683, 'fraction_numerical': 0.014640948175691379, 'mean_word_length': 5.61136712749616, 'pattern_counts': {'":': 0, '<': 30, '<?xml version=': 0, '>': 30, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '222397', 'license': 'MIT', 'n_tokens_mistral': 1319, 'n_tokens_neox': 1246, 'n_words': 510, 'path': 'content/ML/ML_4.ipynb', 'repo_name': 'mm5110/PIC16A', 'size': '4312'}