<jupyter_start><jupyter_text>ML Course, Bogotá, Colombia  (&copy; <NAME>; June 2019)<jupyter_code>%run ../talktools.py<jupyter_output><empty_output><jupyter_text>Statistical Learning and Prediction   Machine Learning with scikit-learnMake sure you have version >= 0.19 installed```conda update scikit-learn -y or... pip install scikit-learn```See https://scikit-learn.org/stable/also: also: `mlpy`, `orange`, `keras`, `nolearn`, `tensorflow`, `astroML`, …    Supervised Learning: RegressionUse training set of $(\vec x,y)$ pairs to learn to predict $y$ for new $\vec x$**Regression** - predicting *continuous* outcome ($y$) variable from a vector of input features ($\vec x$)- Linear Regression:  `linear_model.LinearRegression`- Lasso & Ridge Reg.:  `linear_model.Lasso` / `linear_model.Ridge`- Gaussian Process Regression: `gaussian_process.GaussianProcess`- Nearest Neighbor Regression:  `neighbors.KNeighborsRegressor`- Support Vector Regression:   `svm.SVR`- Regression Trees:  `tree.DecisionTreeRegressor`  Regression  Let's take a look at the famous Boston Housing data<jupyter_code>%matplotlib inline
import matplotlib.pyplot as plt
from sklearn import datasets

boston = datasets.load_boston() # Boston house-prices
X = boston['data']   # 13 features (e.g. crime, # rooms, age, etc.)
Y = boston['target'] # response (median house price)
print("feature vector shape=", X.shape)
print("class shape=", Y.shape)
print(boston.feature_names)
print(type(boston.feature_names))
print(boston.DESCR)<jupyter_output><empty_output><jupyter_text>Let's plot some data (Y axis is what we want to predict)<jupyter_code>f,(ax1,ax2,ax3) = plt.subplots(1,3,figsize=(12,6))
ax1.scatter(X[:,11],Y)
ax2.scatter(X[:,10],Y)
ax3.scatter(X[:,9],Y)
from matplotlib import colors
from matplotlib import cm
import matplotlib
import numpy as np
import seaborn as sns
import pandas as pd

sns.set(style="white")
sns.set_context("poster")
fig = plt.figure(figsize=(7,7))
df = pd.DataFrame(X,columns=boston.feature_names)
df["target"]  = boston['target']
col = cm.viridis(Y/max(Y))
g= sns.pairplot(df[['AGE','DIS','RAD','TAX']], 
                plot_kws=dict(s=80*(Y/max(Y))**3,  color=col, edgecolor=None, alpha=0.7))

# make a colorbar
cmap = matplotlib.cm.ScalarMappable(
      norm = colors.Normalize(min(Y), min(Y)), 
      cmap = plt.get_cmap('viridis'))

cmap.set_array([]) # or alternatively cmap._A = []
fig.colorbar(cmap, cax = fig.add_axes([.98, .4, .01, .2]))<jupyter_output><empty_output><jupyter_text>Basic Model FittingWe need to create a **training set** and a **testing set**.<jupyter_code># half of data
import math
half = math.floor(len(Y)/2)
train_X = X[:half]
train_Y = Y[:half]
test_X = X[half:]
test_Y = Y[half:]<jupyter_output><empty_output><jupyter_text>** Linear Regression **The following are a set of methods intended for regression in which the target value is expected to be a linear combination of the input variables. In mathematical notion, if $\hat{y}$ is the predicted value.$$\hat{y}(w, x) = w_0 + w_1 x_1 + ... + w_p x_p$$Across the module, we designate the vector $w = (w_1,..., w_p)$ as `coef_` and $w_0$ as `intercept_`.To perform classification with generalized linear models, see Logistic regression.http://scikit-learn.org/stable/modules/linear_model.html<jupyter_code>from sklearn import linear_model
clf = linear_model.LinearRegression()
clf.
# fit the model
clf.fit(train_X, train_Y)
# now do the prediction
Y_lr_pred = clf.predict(test_X)

# how well did we do?
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(test_Y,Y_lr_pred) ; print(mse)
f, ax = plt.subplots()
ax.plot(test_Y,Y_lr_pred - test_Y,'o')
ax.set_title("Linear Regression Residuals - MSE = %.1f" % mse)
ax.set_xlabel("True Median House Price ($1,000)")
ax.set_ylabel("Residual")
ax.hlines(0,min(test_Y),max(test_Y),color="red")<jupyter_output><empty_output><jupyter_text>** *k*-Nearest Neighbor (KNN) Regression **"The principle behind nearest neighbor methods is to find a predefined number of training samples closest in distance to the new point, and predict the label from these. The number of samples can be a user-defined constant (k-nearest neighbor learning), or vary based on the local density of points (radius-based neighbor learning). The distance can, in general, be any metric measure: standard Euclidean distance is the most common choice. Neighbors-based methods are known as non-generalizing machine learning methods, since they simply “remember” all of its training data (possibly transformed into a fast indexing structure such as a Ball Tree or KD Tree.)."http://scikit-learn.org/stable/modules/neighbors.html<jupyter_code>from sklearn import neighbors
from sklearn import preprocessing
# many methods work better on scaled X
X_scaled = preprocessing.scale(X) 
clf1 = neighbors.KNeighborsRegressor(5)
train_X = X_scaled[:half]
test_X = X_scaled[half:]
clf1.fit(train_X,train_Y)
Y_knn_pred = clf1.predict(test_X)
mse = mean_squared_error(test_Y,Y_knn_pred) ; print(mse)

f, ax = plt.subplots()
ax.plot(test_Y, Y_knn_pred - test_Y,'o',alpha=0.4)
ax.set_title("k-NN Residuals - MSE = %.1f" % mse)
ax.set_xlabel("True Median House Price ($1,000)")
ax.set_ylabel("Residual")
ax.hlines(0,min(test_Y),max(test_Y),color="red")<jupyter_output><empty_output><jupyter_text>Let's see what other datasets are readily available...<jupyter_code>datasets.<jupyter_output><empty_output><jupyter_text>Some of these we need to download<jupyter_code>cal_house = datasets.california_housing
cal_data = cal_house.fetch_california_housing()
print(cal_data.DESCR)
cal_data.feature_names
X = cal_data['data'] # 8 features 
Y = cal_data['target'] # response (median house price)
half = math.floor(len(Y)/2)
len(Y)
from sklearn import neighbors
from sklearn import preprocessing

# remember: many methods work better on scaled X
X_scaled = preprocessing.scale(X) 
train_X = X_scaled[:half]
train_Y = Y[:half]
test_X = X_scaled[half:]
test_Y = Y[half:]
clf1 = neighbors.KNeighborsRegressor(15)
clf1.fit(train_X,train_Y)
import seaborn as sns
sns.set_context("paper")

Y_knn_pred = clf1.predict(test_X)
mse = mean_squared_error(test_Y,Y_knn_pred) ; print(mse)

f, ax = plt.subplots(figsize=(10,6))
ax.scatter(test_Y, Y_knn_pred - test_Y,alpha=0.2,edgecolors=None)
ax.set_title("k-NN Residuals - MSE = %.1f" % mse)
ax.set_xlabel("True log normalized Median House Price")
ax.set_ylabel("Residual")
ax.hlines(0,min(test_Y),max(test_Y),color="red")
ax.set_xlim(0,5.1)<jupyter_output><empty_output><jupyter_text>Error Estimation & Model Selection**Q**: How will our model perform on future data?So far, we’ve split the data, using one set to train the model and the other to test its performanceThis train-test strategy avoids over-fitting to the sample on hand, but wastes data & can produce poor error estimates.cf http://scikit-learn.org/stable/tutorial/statistical_inference/model_selection.html model selection: cross-validation- *K-fold CV* - randomly split the training data into K folds.  For each $k=1,...,K$, train model only on the data not in fold $k$ & predict for data in fold $k$.  Compute performance metric over CV predictions.- *Leave-one-out (LOO) CV* - K-fold CV with  K = number of training points.<jupyter_code>from sklearn import model_selection
model_selection.GridSearchCV # hyperparameter optimization
import numpy as np
from sklearn import datasets

boston = datasets.load_boston()
X = boston['data'] ; y = boston['target']

from sklearn import linear_model
clf = linear_model.LinearRegression()

from sklearn.model_selection import cross_val_score

def print_cv_score_summary(model, xx, yy, cv):
    scores = cross_val_score(model, xx, yy, cv=cv, n_jobs=1)
    print("mean: {:3f}, stdev: {:3f}".format(
        np.mean(scores), np.std(scores)))
# Returns the coefficient of determination R^2 of the prediction.
print_cv_score_summary(clf, X, y,
                       cv=model_selection.KFold(5, shuffle=True, random_state = 42))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.125125250501002, 'fraction_numerical': 0.01089679358717435, 'mean_word_length': 5.8247863247863245, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 39, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '267014', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 2844, 'n_tokens_neox': 2747, 'n_words': 757, 'path': 'Lectures/3_SupervisedLearning/00_Supervised_Learning_regression.ipynb', 'repo_name': 'julianfbaquero/LML', 'size': '7988'}