<jupyter_start><jupyter_text>Analysing model capacityAuthor: <NAME>, based on materials from <NAME><jupyter_code>%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>The issues associated with validation and cross-validation are some of the most importantaspects of the practice of machine learning. Selecting the optimal modelfor your data is vital, and is a piece of the problem that is not oftenappreciated by machine learning practitioners.Of core importance is the following question:**If our estimator is underperforming, how should we move forward?**- Use simpler or more complicated model?- Add more features to each observed data point?- Add more training samples?The answer is often counter-intuitive.  In particular, **sometimes using amore complicated model will give _worse_ results.**  Also, **sometimes addingtraining data will not improve your results.**  The ability to determinewhat steps will improve your model is what separates the successful machinelearning practitioners from the unsuccessful.  Learning Curves and Validation Curves One way to address this issue is to use what are often called **Learning Curves**.The idea is to test the performance of your prediction pipeline as a function of the training samples.To get a "robust" estimation of the model performance, the performance is evaluted multiple times for multiple random splits of the data.What the right model for a dataset is depends critically on **how much data we have**. More data allows us to be more confident about building a complex model. Lets built some intuition on why that is. Look at the following datasets:<jupyter_code>from sklearn import model_selection

rng = np.random.RandomState(0)
n_samples = 100

def f(X):
    return X ** 3

X = np.sort(2. * (rng.rand(n_samples) - .5))
y = f(X) + .01 * rng.randn(n_samples)
X = X[:, None]
y = y

def plot(clf=None):
    fig, axarr = plt.subplots(1, 3, figsize=(12, 4))
    for ax, decim in zip(axarr, [45, 10, 1]):
        ax.scatter(X[::decim], y[::decim])
        ax.set_xlim((-3 * .5, 3 * .5))
        ax.set_ylim((-1, 1))
        xx = np.linspace(X.min(), X.max(), 100)
        ax.plot(xx, f(xx), 'r--')

        if clf is not None:
            xx = np.linspace(-1.5, 1.5, 100)
            clf.fit(X[::decim], y[::decim])
            y_pred = clf.predict(xx[:, None])
            ax.plot(xx, y_pred, 'g')
    plt.show()

plot()<jupyter_output><empty_output><jupyter_text>They all come from the same underlying process. But if you were asked to make a prediction, you would be more likely to draw a straight line for the left-most one, as there are only very few datapoints, and no real rule is apparent. For the dataset in the middle, some structure is recognizable, though the exact shape of the true function is maybe not obvious. With even more data on the right hand side, you would probably be very comfortable with drawing a curved line with a lot of certainty.<jupyter_code>from sklearn.linear_model import LinearRegression

plot(LinearRegression())
from sklearn.svm import SVR 
plot(SVR(kernel='rbf', gamma='auto'))<jupyter_output><empty_output><jupyter_text>A great way to explore how a model fit evolves with different dataset sizes are learning curves.A learning curve plots the validation error for a given model against different training set sizes.But first, take a moment to think about what we're going to see:**Questions:**- **As the number of training samples are increased, what do you expect to see for the training error?  For the validation error?**- **Would you expect the training error to be higher or lower than the validation error?  Would you ever expect this to change?**We can run the following code to plot the learning curve for a ``kernel = linear`` model:<jupyter_code>from sklearn.model_selection import learning_curve
from sklearn.svm import SVR

scoring = 'neg_mean_squared_error'

training_sizes, train_scores, test_scores = learning_curve(SVR(kernel='linear'), X, y, cv=10,
                                                           scoring=scoring,
                                                           train_sizes=[.6, .7, .8, .9, 1.])

# Use the negative because we want to maximize score
print(train_scores.mean(axis=1))
plt.plot(training_sizes, train_scores.mean(axis=1), label="training scores")
plt.plot(training_sizes, test_scores.mean(axis=1), label="test scores")
plt.legend(loc='best');<jupyter_output><empty_output><jupyter_text>You can see that for the model with ``kernel = linear``, the validation score doesn't really improve as more data is given.Notice that the validation error *generally improves* with a growing training set,while the training error *generally gets worse* with a growing training set.  Fromthis we can infer that as the training size increases, they will converge to a singlevalue.From the above discussion, we know that `kernel = linear`underfits the data. This is indicated by the fact that both thetraining and validation errors are very poor. When confronted with this type of learning curve,we can expect that adding more training data will not help matters: bothlines will converge to a relatively high error.**When the learning curves have converged to a poor error, we have an underfitting model.**An underfitting model can be improved by:- Using a more sophisticated model (i.e. in this case, increase complexity of the ``kernel`` parameter)- Gather more features for each sample.- Decrease regularization in a regularized model.A underfitting model cannot be improved, however, by increasing the number of trainingsamples (do you see why?)Now let's look at an overfit model:<jupyter_code>from sklearn.svm import SVR
training_sizes, train_scores, test_scores = learning_curve(SVR(kernel='rbf', gamma='auto'), X, y, cv=10,
                                                           scoring=scoring,
                                                           train_sizes=[.6, .7, .8, .9, 1.])

plt.plot(training_sizes, train_scores.mean(axis=1), label="training scores")
plt.plot(training_sizes, test_scores.mean(axis=1), label="test scores")
plt.legend(loc='best')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08167877320419693, 'fraction_numerical': 0.008716707021791767, 'mean_word_length': 3.972712680577849, 'pattern_counts': {'":': 0, '<': 23, '<?xml version=': 0, '>': 23, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '20169', 'license': 'CC0-1.0', 'n_tokens_mistral': 1727, 'n_tokens_neox': 1637, 'n_words': 758, 'path': '07_advanced_sklearn_usage/06_learning_curves.ipynb', 'repo_name': 'HirahTang/datascience_starter_course', 'size': '6216'}