<jupyter_start><jupyter_text>Lecture 21: Model Iteration and Improvement Applied Machine Learning__<NAME>__Cornell Tech  Practical Considerations When Applying Machine LearningSuppose you trained an image classifier with 80% accuracy. What's next? * Add more data?* Train the algorithm for longer?* Use a bigger model?* Add regularization?* Add new features? We will next learn how to prioritize these decisions when applying ML.  Fast & Data-Driven Model IterationThe key to building great ML systems is to be *data-driven*:* First, establish train, dev, and test sets and performance metrics.* Then, build a quick prototype and look at its performance.* Quickly iterate on the model, fixing the *actual errors* you see.This process can compensate for an initial lack of domain expertise.  Part 1: Error AnalysisA crucial part of understanding model performance is to systematically examine its errors.Error analysis is a formal process by which this can be done.  Review: Model Development WorkflowThe machine learning development workflow has three steps:1. __Training:__ Try a new model and fit it on the training set. 2. __Model Selection__: Estimate performance on the development set using metrics. Based on results, try a new model idea in step 1. 3. __Evaluation__: Finally, estimate real-world performance on test set.  Review: Datasets for Model DevelopmentWhen developing machine learning models, it is customary to work with three datasets:* __Training set__: Data on which we train the models.* __Development set__ (validation or hold-out set): Data used for tuning models.* __Test set__: Data for evaluating the final performance of our model.  How to Analyze ModelsHow do we iterate on a model given its results on the develoment set? * Error analysis (this presentation)* Analyzing overfitting and underfitting (later)* Analyzing distributional changes (next lecture)And many more types of analyses!  Prioritizng Model Improvements: An ExampleSuppose you trained an image classifier over animal photos.* You hear anecdotes that some dogs are misclassified as cats.* You have the option to integrate with a commercial dog classifier, which will take a month to implement.* Should you move forward with the integration? The key to this decision is to closely examine *actual performance*. Suppose you need to decide if it's worth fixing a certain type of error.* First, collect ~100 random dev set examples that were *misclassified*.* Look at these examples, and count how many have that error.If 5% of misclassified examples have that problem, it's probably not important. If 50% do, then it's important.  Error AnalysisError analysis systematically identifies the most common errors made by the model. 1. Collect a random subset of misclassified dev set examples.2. Manually examine them; identify the most common types or *categories* of errors.3. Count \% of points affected by each error categories.You should prioritize the most common error categories.  Error CategoriesError analysis involves classifying errors into *categories*.* A category is a systematic error made by the model.* Categories can come from intuition, but ideally they are constructed by manually examining the data.  Error Analysis: An ExampleSuppose you just trained a new image classifier.* After looking at the first 20-30 errors, you realize they are all due to images being either *blurry*, *flipped*, or *mislabeled*.* These become the error categories. We then go through through the random subset of errors and assign them to categories.$ $     | Blurry | Flipped | Mislabeled--------|--------|---------|-----------Image 1 | X      | X       |        Image 2 |        | X       |       Image 3 |        |         | X      ...     |        |         |       __Total__   | 20%    | 50%     | 30%We know that the most important fix is to correct for flipped images.  Mislabeled DataReal-world data is often messy, and labels are not always correct.* __Scenario 1__: 10% error, of which 0.4% is due to labeling and 9.6% to other causes. No need to relabel.* __Scenario 2__: 2% error, of which 0.4% is due to labeling and 1.6% to other causes. Important to relabel!* These errors can be identified as part of error analysis* The prevalence of these errors tells us if we should fix themIt's important to fix labeling issues if they prevent us from measuring model error.  Development Set SizeHow big should the dev set size be? Error analysis suggests a lower bound.* You should have at least ~50-100 mislabeled errors to analyze.* If model has error of $\epsilon$, we want a dev size of ~$100/\epsilon$.    * e.g., for 5% error, we want a dev set of size ~2,000.Also, remember to periodically update the dev set to minimize overfitting.  Error Analysis on the Training SetShould we perform error analysis on the dev set or the training set? * Ultimately, we are interested in errors that will occur in production. This is best estimated on the dev set. * However, if a model makes errors on the training set, it will make them on the dev set as well. Hence, analyzing and fixing training set errors is also important.  Error Analysis: Another ExampleLet's look at another example of error analysis on a small toy dataset. We will use the `sklearn` digits dataset, a downscaled version of MNIST.<jupyter_code>from sklearn.datasets import load_digits
digits = load_digits()<jupyter_output><empty_output><jupyter_text>We can visualize these digits as follows:<jupyter_code>from matplotlib import pyplot as plt

plt.figure(figsize=(8,16))
_, axes = plt.subplots(2, 5)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes.flatten(), images_and_labels[:10]):
    ax.set_axis_off()
    ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    ax.set_title('Digit %i' % label)<jupyter_output><empty_output><jupyter_text>Let's separate this data into two equal-sized training and dev sets.<jupyter_code># https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html
from sklearn.model_selection import train_test_split

# Split data into train and test subsets
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))
X_train, X_dev, y_train, y_dev = train_test_split(
    data, digits.target, test_size=0.5, shuffle=False)<jupyter_output><empty_output><jupyter_text>We can train a simple Softmax model on this data.<jupyter_code># https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html
from sklearn.linear_model import LogisticRegression
classifier = LogisticRegression(max_iter=1e7)

# We learn the digits on the first half of the digits
classifier.fit(X_train, y_train)

# Now predict the value of the digit on the second half:
predicted = classifier.predict(X_dev)<jupyter_output><empty_output><jupyter_text>It achieves the following accuracy.<jupyter_code>(predicted == y_dev).mean()<jupyter_output><empty_output><jupyter_text>We hypothesize that certain digits are misclassified more than others.<jupyter_code># these dev set digits are classified incorrectly
X_error = X_dev[predicted != y_dev]
y_error = y_dev[predicted != y_dev]
p_error = predicted[predicted != y_dev]

# these dev set digits are classified correctly
X_corr = X_dev[predicted == y_dev]
y_corr = y_dev[predicted == y_dev]
p_corr = predicted[predicted == y_dev]

# show the histogram
plt.xticks(range(10))
plt.hist(y_error)<jupyter_output><empty_output><jupyter_text>The most common misclassified digit is a 3.We can investigate the issue by looking at a subset of misclassified 3's (top row) and compare them to correctly classified 3's (bottom row).<jupyter_code>plt.figure(figsize=(8,16))
_, axes = plt.subplots(2, 8)

# these images are classified incorrectly
images_and_labels = list(zip(X_error[y_error==3], p_error[y_error==3]))
for ax, (image, label) in zip(axes[0,:], images_and_labels[:8]):
    ax.set_axis_off()
    ax.imshow(image.reshape((8,8)), cmap=plt.cm.gray_r, interpolation='nearest')
    ax.set_title('f(x)=%i' % label)

# these images are classified correctly
images_and_labels = list(zip(X_corr[y_corr==3], p_corr[y_corr==3]))    
for ax, (image, label) in zip(axes[1,:], images_and_labels[:8]):
    ax.set_axis_off()
    ax.imshow(image.reshape((8,8)), cmap=plt.cm.gray_r, interpolation='nearest')
    ax.set_title('f(x)=%i' % label)<jupyter_output><empty_output><jupyter_text>We discover that the model is misclassifying a particular stlye of 3, and we can focus our efforts on this type of error.  Limitations of Error AnalysisThe main limitations of error analysis include:1. It is particularly easy to overfit the dev set, since we prioritize fixing dev set errors.2. It can be laborious (but still important!)3. Certain bigger trends (e.g. overfitting the data) may be less obvious.Hence, we perform other analyses to explain and diagnose errors.  Part 2: Bias/Variance AnalysisAnother way to understand the performance of the model is to examine the extent to which it's overfitting or underfitting the data. We refer to this as bias/variance analysis.  Review: Error AnalysisError analysis systematically identifies the most common errors made by the model. 1. Collect a random subset of misclassified dev set examples.2. Manually examine them; identify the most common types or *categories* of errors.3. Count \% of points affected by each error categories.You should prioritize the most common error categories.  Review: Overfitting (Variance)Overfitting is one of the most common failure modes of machine learning.* A very expressive model (a high degree polynomial) fits the training dataset perfectly.* The model also makes wildly incorrect prediction outside this dataset, and doesn't generalize. Recall this example, where we randomly sample around a true function.<jupyter_code>import numpy as np
np.random.seed(1)
n_samples = 40
true_fn = lambda X: np.cos(1.5 * np.pi * X)

X = np.sort(np.random.rand(n_samples))
y = true_fn(X) + np.random.randn(n_samples) * 0.1

X_line = np.linspace(0, 1, 100)
plt.plot(X_line, true_fn(X_line), label="True function")
plt.scatter(X, y, edgecolor='b', s=20, label="Samples")<jupyter_output><empty_output><jupyter_text>Below, we fit a high degree polynomial on random samples of 30 points from this dataset.Each small subset of the data that we train on results is a very different model.<jupyter_code>from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

n_plots, X_line = 3, np.linspace(0,1,20)
plt.figure(figsize=(14, 5))
for i in range(n_plots):
    ax = plt.subplot(1, n_plots, i + 1)
    random_idx = np.random.randint(0, 30, size=(30,))
    X_random, y_random = X[random_idx], y[random_idx]

    polynomial_features = PolynomialFeatures(degree= z, include_bias=False)
    linear_regression = LinearRegression()
    pipeline = Pipeline([("pf", polynomial_features), ("lr", linear_regression)])
    pipeline.fit(X_random[:, np.newaxis], y_random)

    ax.plot(X_line, true_fn(X_line), label="True function")    
    ax.plot(X_line, pipeline.predict(X_line[:, np.newaxis]), label="Model")
    ax.scatter(X_random, y_random, edgecolor='b', s=20, label="Samples", alpha=0.2)
    ax.set_xlim((0, 1))
    ax.set_ylim((-2, 2))
    ax.legend(loc="best")
    ax.set_title('Random sample %d' % i)<jupyter_output><empty_output><jupyter_text>An algorithm that has a tendency to overfit is also called __high variance__, because it outputs a predictive model that varies a lot if we slightly perturb the dataset.  Review: Underfitting (Bias)Underfitting is another common problem in machine learning.* The model is too simple to fit the data well (e.g., approximating a high degree polynomial with linear regression).* As a result, the model is not accurate on training data and is not accurate on new data.Because the model cannot fit the data, we say it's __high bias__. We may compare overfitting vs underfitting on our polynomial dataset.<jupyter_code>degrees = [1, 20, 5]
titles = ['Underfitting', 'Overfitting', 'A Good Fit']
plt.figure(figsize=(14, 5))
for i in range(len(degrees)):
    ax = plt.subplot(1, len(degrees), i + 1)

    polynomial_features = PolynomialFeatures(degree=degrees[i], include_bias=False)
    linear_regression = LinearRegression()
    pipeline = Pipeline([("pf", polynomial_features), ("lr", linear_regression)])
    pipeline.fit(X[:, np.newaxis], y)

    ax.plot(X_test, true_fn(X_test), label="True function")    
    ax.plot(X_test, pipeline.predict(X_test[:, np.newaxis]), label="Model")
    ax.scatter(X, y, edgecolor='b', s=20, label="Samples", alpha=0.2)
    ax.set_xlim((0, 1))
    ax.set_ylim((-2, 2))
    ax.legend(loc="best")
    ax.set_title("{} (Degree {})".format(titles[i], degrees[i]))<jupyter_output><empty_output><jupyter_text>On the Signficance of Bias and VarianceEvery error in machine learning is either underfitting (bias) or overfitting (variance).* __Bias__: Our model is not able to extract signal from data.* __Variance__: The signal doesn't generalize to new data.By definition, if we have no bias and variance, we have a perfect model. Hence, bias/variance is important to understand.  Quantifying Bias and VarianceWe approximately quantify the bias and the variance of a model as follows.$$\text{dev error} = (\underbrace{\text{dev error} - \text{train error}}_\text{variance}) + \underbrace{\text{train error}}_\text{bias}$$ * The bias is simply the training error (can we even fit training data?) * Variance is the extra error on the dev set (do we generalize to it?) It's important to consider both types of errors.* If a model has high training error, it will certainly have high generalization error.* As we fix training error, we need to ensure that it generalizes to new data.We can make different changes to the algorithm to address both of these issues.  Diagnosing Bias and VarianceWe may use this observation to diagnose bias/variance in practice.* If the training error is high, we have high bias.* If dev error is much higher than training error, we have high variance. Consider the following example:* Our training set error is 15%.* Our dev set error is 16%.* Human error is 2%.This is a typical example of __high bias__ (underfitting). Next, consider another example:* Our training set error is 4%.* Our dev set error is 12%.* Human error is 2%.This is an example of __high variance__ (overfitting). Finally, suppose you see the following:* Our training set error is 2.5%.* Our dev set error is 3%.* Human error is 2%.This is a model that seems to work quite well!  Addressing VarianceThe best way to reduce variance is to __give the model more data__. * Overfitting occurs because we don't have enough data to fit an expressive model.* Hence, we add data and the memorized solution is no longer good.However, this may be not be feasible because of high costs for compute or data acquisition. Alternative options for reducing variance include:* __Add Regularization__: L2 regularization, dropout, early stopping (increases bias)* __Drop or Change Features__: Ideally, drop irrelevant ones (needs manual analysis)* __Reduce Model Size__: Decrease layer size, kernel complexity, etc. Regularization often works better.  Addressing BiasThe best way to reduce bias is to __increase the expressivity of the model__. * e.g., add more neural net layers, decrease kernel bandwidth, increase polynomial degree.* A bigger (more expressive) model will fit the data better.However, this may be not be feasible because of high costs for compute. Alternative options for reducing bias include:* __Add or Change Features__: Often based on manual analysis.* __Reduce Regularization__: Change dropout, train model longer (incrases variance).* __Adjust Model Architecture__: e.g., switch to ConvNets for image classification, which are more parameter-efficient than fully-connected networks (can also improve variance) For both bias and variance reduction, we can use error analysis to guide our changes, e.g.:* To determine which features to add/remove.* To determine which additional data to collect.  Bias/Variance Analysis: An ExampleLet's use our earlier example with the sklearn digits dataset to illustrate this approach. Recall our digits dataset from earlier:<jupyter_code>from matplotlib import pyplot as plt

plt.figure(figsize=(8,16))
_, axes = plt.subplots(2, 5)
images_and_labels = list(zip(digits.images, digits.target))
for ax, (image, label) in zip(axes.flatten(), images_and_labels[:10]):
    ax.set_axis_off()
    ax.imshow(image, cmap=plt.cm.gray_r, interpolation='nearest')
    ax.set_title('Digit %i' % label)<jupyter_output><empty_output><jupyter_text>We can train a small fully-connected neural network on this data.<jupyter_code># https://scikit-learn.org/stable/auto_examples/classification/plot_digits_classification.html
from sklearn.neural_network import MLPClassifier
classifier = MLPClassifier()

# We learn the digits on the first half of the digits
classifier.fit(X_train, y_train)

# Now predict the value of the digit:
predicted = classifier.predict(X_dev)
predicted_train = classifier.predict(X_train)<jupyter_output><empty_output><jupyter_text>It achieves the following accuracy.<jupyter_code>print('Training set accuracy: %.3f ' % (predicted_train == y_train).mean())
print('Development set accuracy: %.3f ' % (predicted == y_dev).mean())<jupyter_output>Training set accuracy: 1.000 
Development set accuracy: 0.937<jupyter_text>We have clearly memorized our dataset, and are overfitting. Let's increase regularization.<jupyter_code>classifier = MLPClassifier(max_iter=1000, alpha=1)

# We learn the digits on the first half of the digits
classifier.fit(X_train, y_train)

# Now predict the value of the digit:
predicted = classifier.predict(X_dev)
predicted_train = classifier.predict(X_train)<jupyter_output><empty_output><jupyter_text>By increasing L2 regularization (`alpha`), we improve performance by 1%.(Although we still somewhat overfit)<jupyter_code>print('Training set accuracy: %.3f ' % (predicted_train == y_train).mean())
print('Development set accuracy: %.3f ' % (predicted == y_dev).mean())<jupyter_output>Training set accuracy: 1.000 
Development set accuracy: 0.947
{'fraction_non_alphanumeric': 0.08521138345122553, 'fraction_numerical': 0.011131216757142074, 'mean_word_length': 5.075283144570286, 'pattern_counts': {'":': 0, '<': 60, '<?xml version=': 0, '>': 60, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '119027', 'license': 'Apache-2.0', 'n_tokens_mistral': 5364, 'n_tokens_neox': 4997, 'n_words': 2199, 'path': '_site/notes/notes_ipynb/lecture21-model-iteration.ipynb', 'repo_name': 'wu-hongjun/aml', 'size': '18249'}