<jupyter_start><jupyter_text>*This notebook is part of  course materials for CS 345: Machine Learning Foundations and Practice at Colorado State University.Original versions were created by <NAME>.The content is availabe [on GitHub](https://github.com/asabenhur/CS345).**The text is released under the [CC BY-SA license](https://creativecommons.org/licenses/by-sa/4.0/), and code is released under the [MIT license](https://opensource.org/licenses/MIT).*<jupyter_code>%autosave 0
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Nonlinear modeling using basis function regresssion A linear function is not sufficient for fitting data that does not follow a linear relationship:<jupyter_code>from sklearn.linear_model import LinearRegression
lr = LinearRegression()

# create data
rng = np.random.RandomState(42)
N = 30
x = 10 * rng.rand(N)
y = np.sin(x) + 0.2 * rng.randn(N)

xfit = np.linspace(0, 10, 100)

lr.fit(x[:, np.newaxis], y)
yfit = lr.predict(xfit[:, np.newaxis])

plt.scatter(x, y, alpha=0.5, label='data')
plt.plot(xfit, yfit, label='linear fit');
plt.plot(xfit, np.sin(xfit), label = 'sin(x)')
plt.legend();<jupyter_output><empty_output><jupyter_text>There are two ways to address this issue:* Use a non-linear classifier* Transform the features using a non-linear function.We will explore the latter approach.First, consider data with a single feature $x$.  Instead of using $x$ as-is, let us first transform it using a collection of nonlinear functions.  As an example, replace $x$ with all monomials of degree up to $D$:$$(x^0, x^1, x^2, x^3, \ldots ,x^D).$$We will use a linear model using these features:$$f(x) = w_0 + w_1 x + w_2 x^2 + \ldots, + w_D x^D.$$The model is still linear in the parameters, even though the function it computes is nonlinear.  Therefore our machinery for computing the solution still works without any need for modification - all we need to do is transform our features.<jupyter_code>from sklearn.preprocessing import PolynomialFeatures

X = np.array([[2],[3]])
poly = PolynomialFeatures(3, include_bias=True)
poly.fit_transform(X)<jupyter_output><empty_output><jupyter_text>Let's apply this idea and see how well it works:<jupyter_code>from sklearn.pipeline import make_pipeline
poly_model = make_pipeline(PolynomialFeatures(6),
                           LinearRegression())
rng = np.random.RandomState(42)
N = 30
x = 10 * rng.rand(N)
y = np.sin(x) + 0.2 * rng.randn(N)

xfit = np.linspace(0, 10, 100)
poly_model.fit(x[:, np.newaxis], y)
yfit = poly_model.predict(xfit[:, np.newaxis])

plt.scatter(x, y, alpha=0.5)
plt.plot(xfit, yfit);<jupyter_output><empty_output><jupyter_text>The code we used above used a new scikit-learn concept - **pipelines**.In many cases we need to apply several steps of *preprocessing* to our data before applying the *fit* function.  These steps need to be applied to both training and test data.  To simplify the process we can express these steps as a pipeline that encapsulates the series of steps, and performs them all automatically on the input data.  These steps can include many types of feature manipulation and processing:* Standardization* Selection of relevant features* Compute nonlinear featuresPipelines make the preprocessing steps as part of the classifier.Pipelines will support the process of selecting parameters of the classification process (e.g. degree of nonlinear features or the number of features that are to be used).  This will allow treating these parameters as hyperparameters of your classification pipeline.  ExerciseExplore the above code as a function of the degree of the polynomial.What are you observing?  Nonlinear features for higher dimensional dataSo far we explored nonlinear features in one dimension.  Let's explore the idea for higher dimensional data.In one dimension we mapped our data to the set of monomials of up to degree $D$:$$(1, x^1, x^2, x^3, \ldots ,x^D).$$To generalize this to a $d$ dimensional feature vector $\mathbf{x}$, we consider all monomials with degree up to $D$.  For example, for two dimensional data $\mathbf{x} = (x_1, x_2)$ and $D=2$ the resulting set of features is$$(1, x_1, x_2, x_1^2, x_2^2, x_1 x_2)$$and our regression function takes on the form:$$f(\mathbf{x}) = w_0 + w_1 x_1 + w_2 x_2 + w_3 x_1^2 + w_4 x_2^2 + w_5 x_1 x_2$$And in scikit-learn:<jupyter_code>from sklearn.preprocessing import PolynomialFeatures

X = np.array([[2, 3], [3, 4]])

poly = PolynomialFeatures(2, include_bias=True)
# include bias determines if to include the case where all feature 
# powers are zero; this serves as a bias term 
poly.fit_transform(X)<jupyter_output><empty_output><jupyter_text>Questions* Write out the set of monomials up to degree $3$ for two dimensional data* Assuming your data is in $d$ dimensions, what is the dimensionality of the resulting features when using monomials of degree up to 2?  What does that tell us about the feasibility of this approach for high dimensional data? The approach described above is regression using **polynomial basis functions**.  More generally we can express this approach as creating a transformation of the data using a set of functions known as *basis functions*:$$\Phi(\mathbf{x}) = (\phi_1(\mathbf{x}),\ldots,\phi_D(\mathbf{x}))$$Another common approach is to use Gaussian rather than polynomial functions as your basis functions.  OverfittingWhile mapping our data to a higher dimensional feature space is very useful for being able to model data that exhibit non-linear relationships, that increased flexibility can lead to dangerous consequences as the next figure demonstrates:<jupyter_code># code based on https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression

def true_fun(X):
    return np.cos(1.5 * np.pi * X)

np.random.seed(0)

n_samples = 30
degrees = [1, 4, 15]

X = np.sort(np.random.rand(n_samples))
y = true_fun(X) + np.random.randn(n_samples) * 0.1

plt.figure(figsize=(14, 5))
for i in range(len(degrees)):
    ax = plt.subplot(1, len(degrees), i + 1)
    plt.setp(ax, xticks=(), yticks=())

    polynomial_features = PolynomialFeatures(degree=degrees[i],
                                             include_bias=False)
    linear_regression = LinearRegression()
    pipeline = Pipeline([("polynomial_features", polynomial_features),
                         ("linear_regression", linear_regression)])
    pipeline.fit(X[:, np.newaxis], y)

    Xs = np.linspace(0, 1, 100)
    plt.plot(Xs, pipeline.predict(Xs[:, np.newaxis]), label="Model")
    plt.plot(Xs, true_fun(Xs), label="True function")
    plt.scatter(X, y, edgecolor='b', s=30, alpha=0.7,label="Training data")
    plt.xlabel("x")
    plt.ylabel("y")
    plt.xlim((0, 1))
    plt.ylim((-2, 2))
    plt.legend(loc="best")
    plt.title("Degree {}".format(degrees[i]))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10380179410508329, 'fraction_numerical': 0.01708671507902606, 'mean_word_length': 4.829045643153527, 'pattern_counts': {'":': 0, '<': 26, '<?xml version=': 0, '>': 26, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '459715', 'license': 'MIT', 'n_tokens_mistral': 2233, 'n_tokens_neox': 2146, 'n_words': 797, 'path': 'notebooks/module04_01_overfitting_basis_function_regression.ipynb', 'repo_name': 'lottieandrews/CS345', 'size': '7028'}