<jupyter_start><jupyter_text>Supervised Learning Part 1 -- Regression Analysis In regression we are trying to predict a continuous output variable -- in contrast to the nominal variables we will be predicting in the classification examples later. Let's start with a simple toy example with one feature dimension (explanatory variable) and one target variable. We will create a dataset out of a sinus curve with some noise:<jupyter_code>x = np.linspace(-3, 3, 100)
print(x)
rng = np.random.RandomState(42)
y = np.sin(4 * x) + x + rng.uniform(size=len(x))
plt.plot(x, y, 'o');<jupyter_output><empty_output><jupyter_text>Linear RegressionThe first model that we will introduce is the so-called simple linear regression. Here, we want to fit a line to the data.One of the simplest models is a linear one, that simply tries to predict the data as lying on a line. One way to find such a line is `LinearRegression` (also known as [*Ordinary Least Squares (OLS)*] (https://en.wikipedia.org/wiki/Ordinary_least_squares) regression). The scikit-learn API requires us to provide the target variable (`y`) as a 1-dimensional array; scikit-learn's API expects the samples (`X`) in form a 2-dimensional array -- even though it may only consist of 1 feature. Thus, let us convert the 1-dimensional `x` NumPy array into an `X` array with 2 axes:<jupyter_code>print('Before: ', x.shape)
X = x[:, np.newaxis]
print('After: ', X.shape)<jupyter_output><empty_output><jupyter_text>Regression is a supervised task, and since we are interested in its performance on unseen data, we split our data into two parts:1. a training set that the learning algorithm uses to fit the model2. a test set to evaluate the generalization performance of the modelThe ``train_test_split`` function from the ``model_selection`` module does that for us -- we will use it to split a dataset into 75% training data and 25% test data. We start by splitting our dataset into a training (75%) and a test set (25%):<jupyter_code>from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)<jupyter_output><empty_output><jupyter_text>The scikit-learn estimator API and Logistic Regresion  Every algorithm is exposed in scikit-learn via an ''Estimator'' object. (All models in scikit-learn have a very consistent interface). For instance, we first import the linear regression class.<jupyter_code>from sklearn.linear_model import LinearRegression<jupyter_output><empty_output><jupyter_text>Next, we use the learning algorithm implemented in `LinearRegression` to **fit a regression model to the training data**:<jupyter_code>regressor = LinearRegression()
regressor.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>(Some estimator methods such as `fit` return `self` by default. Thus, after executing the code snippet above, you will see the default parameters of this particular instance of `LinearRegression`. Another way of retrieving the estimator's ininitialization parameters is to execute `regressor.get_params()`, which returns a parameter dictionary.) After fitting to the training data, we paramerterized a linear regression model with the following values.<jupyter_code>print('Weight coefficients: ', regressor.coef_)
print('y-axis intercept: ', regressor.intercept_)<jupyter_output><empty_output><jupyter_text>Since our regression model is a linear one, the relationship between the target variable (y) and the feature variable (x) is defined as:$$y = weight \times x + \text{intercept}$$Plugging in the min and max values into thos equation, we can plot the regression fit to our training data:<jupyter_code>min_pt = X.min() * regressor.coef_[0] + regressor.intercept_
max_pt = X.max() * regressor.coef_[0] + regressor.intercept_

plt.plot([X.min(), X.max()], [min_pt, max_pt])
plt.plot(X_train, y_train, 'o');<jupyter_output><empty_output><jupyter_text>Similar to the estimators for classification in the previous notebook, we use the `predict` method to predict the target variable. And we expect these predicted values to fall onto the line that we plotted previously:<jupyter_code>y_pred_train = regressor.predict(X_train)
plt.plot(X_train, y_train, 'o', label="data")
plt.plot(X_train, y_pred_train, 'o', label="prediction")
plt.plot([X.min(), X.max()], [min_pt, max_pt], label='fit')
plt.legend(loc='best')<jupyter_output><empty_output><jupyter_text>As we can see in the plot above, the line is able to capture the general slope of the data, but not many details. Next, let's try the test set:<jupyter_code>y_pred_test = regressor.predict(X_test)
plt.plot(X_test, y_test, 'o', label="data")
plt.plot(X_test, y_pred_test, 'o', label="prediction")
plt.plot([X.min(), X.max()], [min_pt, max_pt], label='fit')
plt.legend(loc='best');<jupyter_output><empty_output><jupyter_text>There is also a convenience function , ``score``, that all scikit-learn classifiers have to compute how good the model is:   For regression tasks, this is the **R2** score. $$R^2= 1- \frac{\sum_{i}(f_i-y_i)^2}{\sum_{i}(y_i-\mu)^2}$$<jupyter_code>regressor.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>___ ExerciseWe will now look at a dataset with more than one variable. The ideas from the simple example above can be reused (we do not have to add an additional axis to the data).Create a linear regressor that fits the ``diabetes`` data.The methods that plot the data above cannot be used directly, can you adapt them in some way?<jupyter_code>from sklearn.datasets import load_diabetes

diabetes_data = load_diabetes()
X, y = diabetes_data.data, diabetes_data.target
print("The data is described at https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html. The short description is:")
print("Ten baseline variables, age, sex, body mass index, average blood pressure, and six blood serum measurements were obtained for each of n = 442 diabetes patients, as well as the response of interest, a quantitative measure of disease progression one year after baseline.")

# We need to:
# Split the data into training and test sets
# Create a LinearRegression object
# Fit the regressor object to the training data
# Evaluate the model on the test data


# ...
from sklearn.decomposition import PCA
pca = PCA(n_components=1).fit(X)
X_decomp = pca.transform(X_test)
regressor.fit(X_train, y_train)
plt.plot(X_decomp, y_test, 'o')
plt.plot(X_decomp, regressor.predict(X_test), 'x')
X_noise = np.random.randn(*X_train.shape)
regressor.fit(X_noise, y_train)
plt.plot(X_decomp, y_test, 'o')
plt.plot(X_decomp, regressor.predict(X_test), 'x')<jupyter_output><empty_output><jupyter_text>___  KNeighborsRegression This is a simple regression method that: given a new, unknown observation, look up in your reference database which ones have the closest features and take the output of the nearest point, or we could average several nearest points. This method is less popular for regression than for classification, but still a good baseline.<jupyter_code># First recreate our synthetic data
x = np.linspace(-3, 3, 100)
rng = np.random.RandomState(42)
y = np.sin(4 * x) + x + rng.uniform(size=len(x))
X = x[:, np.newaxis]

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
from sklearn.neighbors import KNeighborsRegressor
kneighbor_regression = KNeighborsRegressor(n_neighbors=1)
kneighbor_regression.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Again, let us look at the behavior on training and test set:<jupyter_code>y_pred_train = kneighbor_regression.predict(X_train)

plt.plot(X_train, y_train, 'o', label="data", markersize=10)
plt.plot(X_train, y_pred_train, 's', label="prediction", markersize=4)
plt.legend(loc='best');<jupyter_output><empty_output><jupyter_text>On the training set, we do a perfect job: each point is its own nearest neighbor!<jupyter_code>y_pred_test = kneighbor_regression.predict(X_test)

plt.plot(X_test, y_test, 'o', label="data", markersize=8)
plt.plot(X_test, y_pred_test, 's', label="prediction", markersize=4)
plt.legend(loc='best');<jupyter_output><empty_output><jupyter_text>On the test set, we also do a better job of capturing the variation, but our estimates look much messier than before.Let us look at the R2 score:<jupyter_code>kneighbor_regression.score(X_test, y_test)<jupyter_output><empty_output><jupyter_text>Much better than before! Here, the linear model was not a good fit for our problem; it was lacking in complexity and thus under-fit our data. As with linear regression, use the diabetes dataset and create a KNN-regressor.<jupyter_code>from sklearn.datasets import load_diabetes

diabetes_data = load_diabetes()
X, y = diabetes_data.data, diabetes_data.target
print("The data is described at https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html. The short description is:")
print('"Ten baseline variables, age, sex, body mass index, average blood pressure, and six blood serum measurements were obtained for each of n = 442 diabetes patients, as well as the response of interest, a quantitative measure of disease progression one year after baseline."')

# We need to:
# Split the data into training and test sets
# Create a KNeighborsRegressor object (how many neighbours do you want to use?)
# Fit the regressor object to the training data
# Evaluate the model on the test data

# ...
from sklearn.decomposition import PCA
pca = PCA(n_components=1).fit(X)
X_decomp = pca.transform(X_test)
kneighbor_regression.fit(X_train, y_train)
plt.plot(X_decomp, y_test, 'o')
plt.plot(X_decomp, kneighbor_regression.predict(X_test), 'x')<jupyter_output><empty_output><jupyter_text>___ ExerciseCreate a KNN-regressor for the diabetes data.___ ___ ExerciseCompare the KNeighborsRegressor and LinearRegression on the boston housing dataset. You can load the dataset using ``sklearn.datasets.load_boston``. You can learn about the dataset by reading the ``DESCR`` attribute.<jupyter_code># %load solutions/knn_vs_linreg.py<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09586727597854162, 'fraction_numerical': 0.0065567256109676135, 'mean_word_length': 6.160028449502134, 'pattern_counts': {'":': 0, '<': 69, '<?xml version=': 0, '>': 69, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '46048', 'license': 'MIT', 'n_tokens_mistral': 3261, 'n_tokens_neox': 3072, 'n_words': 1134, 'path': 'notebooks/02 - Supervised learning - Part 1.ipynb', 'repo_name': 'fordanic/cmiv-ai-course', 'size': '10066'}