<jupyter_start><jupyter_text>regression in scikit-learn There are three main obstacles to working with regression in scikit-learn:1. Selecting the proper model2. Understanding the proper input data type, and scikit-learn syntax3. Testing and validationToday we will touch on all three, but focus on number 2.  1) Dataset and prep For demonstration, we will use a boston housing dataset, which comes with scikit-learn:<jupyter_code>from sklearn.datasets import load_boston

boston = load_boston()<jupyter_output><empty_output><jupyter_text>If you are going to follow along in other tutorials in the scikit-learn documentation, you will need to know the data structures used as inputs to the models. Let'see what's in the boston dataset:<jupyter_code>boston.keys()<jupyter_output><empty_output><jupyter_text>The description will tell us more about the dataset:<jupyter_code>boston.DESCR<jupyter_output><empty_output><jupyter_text>So we are working on predicitng median value of a home from 506 observations, and 13 covariates including crime rate, lot size, industry/commercial proportion, presence of the Charles River, nitric oxide concentration, rooms per dwelling, units built before 1940, distance to employment centers, access to highways, tax rate, school proxy, black population, and status. To get the variable names we can ask for them in the dictionary:<jupyter_code>print(boston.feature_names)
print()
print(type(boston.feature_names))
print()
print(len(boston.feature_names))<jupyter_output><empty_output><jupyter_text>We see the input is a numpy array of strings for the variable labels. To get the variable data, we ask the dictionary for the data:<jupyter_code>print(boston.data)
print()
print(type(boston.data))
print()
print(len(boston.data))<jupyter_output><empty_output><jupyter_text>The data is a numpy array, inside of which there is a separate array for each observation (all 506 for each hous, *not* 13 for each variable). Each inner array *must* lineup with the order of the variables *and* all other arrays. **ORDER MATTERS** The target, or *y* is accessed in the dictionary as well:<jupyter_code>print(boston.target)
print()
print(type(boston.target))
print()
print(len(boston.target))<jupyter_output><empty_output><jupyter_text>The target array is only one dimension, lined up in order with the with the observations in the data array. Now that we're familiar with the input data, we need to split it up for training and testing:<jupyter_code>from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(boston.data, boston.target,
                                                    train_size=0.75, test_size=0.25)<jupyter_output><empty_output><jupyter_text>Now we have 75% of the data as training data, and 25% of the data as testing data:<jupyter_code>print(len(X_train), len(y_train))
print()
print(len(X_test), len(y_test))<jupyter_output><empty_output><jupyter_text>In scikit-learn, as soon as you have `X_train`, `X_test`, `y_train`, and `y_test`, everything else is just a matter of choosing parameters for whichever model you choose. But this should not be trivialized, selecting models and that model's parameters is *very* important. While we will not cover it here, you should always select the model and parameters best suited for your data.  2) Building models The syntax in scikit-learn does not change for each model, only the parameters. Examples of various models are given below:  Linear Regression We'll start with a basic OLS linear regression model:<jupyter_code>from sklearn import linear_model
lin_reg = linear_model.LinearRegression(n_jobs=1)  # CPUs to use)

model = lin_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>GLM - Ridge Regression<jupyter_code>from sklearn import linear_model
ridge_reg = linear_model.Ridge(alpha=1.0,  # regularization
                               normalize=True,  # normalize X regressors
                               solver='auto')  # options = ‘auto’, ‘svd’, ‘cholesky’, ‘lsqr’, ‘sparse_cg’, ‘sag'

model = ridge_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>GLM - Elastic Net Regression<jupyter_code>elastic_reg = linear_model.ElasticNet(alpha=1.0,  # penalty, 0 is OLS 
                               random_state=None,
                               selection='cyclic')  # or 'random', which converges faster

model = elastic_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>Support Vector Regression<jupyter_code>from sklearn import svm

sv_reg = svm.SVR(kernel='linear',  # ‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’
                 degree=3,  # only used for 'poly' above
                 gamma='auto',  # kernal coeff, default auto is 1/n_features
                 C=1.0)

model = sv_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>K-nearest neighbors regression<jupyter_code>from sklearn import neighbors

knn_reg = neighbors.KNeighborsRegressor(n_neighbors=5,
                                        weights='uniform',  # ‘distance’ weights points by inverse of their distance
                                        algorithm='auto',  # out of ‘ball_tree’, ‘kd_tree’, ‘brute’
                                        leaf_size=30)  # for tree algorithms

model = knn_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>Random Forest Regression<jupyter_code>from sklearn import ensemble

rf_reg = ensemble.RandomForestRegressor(n_estimators=10,  # number of trees
                                        criterion='mse',  # how to measure fit
                                        max_depth=None,  # how deep tree nodes can go
                                        min_samples_split=2,  # samples needed to split node
                                        min_samples_leaf=1,  # samples needed for a leaf
                                        min_weight_fraction_leaf=0.0,  # weight of samples needed for a node
                                        max_features='auto',  # max feats
                                        max_leaf_nodes=None,  # max nodes
                                        n_jobs=1)  # how many to run parallel

model = rf_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>Boosting - AdaBoost Regression<jupyter_code>ab_reg = ensemble.AdaBoostRegressor(base_estimator=None,  # default is DT 
                                    n_estimators=50,  # number to try before stopping
                                    learning_rate=1.0,  # decrease influence of each additional estimator
                                    loss='linear')  # also ‘square’, ‘exponential’


model = ab_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>Neural Network Regression (*new in scikit-learn .18*)<jupyter_code>from sklearn import neural_network
nn_reg = neural_network.MLPRegressor(hidden_layer_sizes=(7, ),  # neurons in ith layer
                                     activation='relu',  # activation function, also 'identity', 'logistic', 'tanh'
                                     solver='adam',  # solves weight optimization, also ‘lbfgs’, ‘sgd’, ‘adam’
                                     alpha=0.0001,  # L2 penalty
                                     batch_size='auto',  # auto is 200
                                     learning_rate='constant',  # schedule for weight updates, also ‘invscaling’, ‘adaptive’
                                     learning_rate_init=0.001,  # lr initial value for sgd and adam
                                     max_iter=200,  # max iterations
                                     momentum=0.9,  # for gradient descent, only sgd
                                     early_stopping=False,  # if validation not improving
                                     validation_fraction=0.1,  # validation subset
                                     beta_1=0.9,  # decay for adam first momentum
                                     beta_2=0.999)  # decay for adam second momentum

model = nn_reg.fit(X_train, y_train)
print(model.score(X_test, y_test))<jupyter_output><empty_output><jupyter_text>3) Cross Validation Our division of training and testing data was naïve, ideally you will do k-fold cross-validation:<jupyter_code>from sklearn.model_selection import cross_val_score
cv_scores = cross_val_score(sv_reg, boston.data, boston.target, cv=3)
print(cv_scores)
import numpy as np

print (np.mean(cv_scores))<jupyter_output><empty_output><jupyter_text>AutoML on top of sklearn AutoML packages still require preparing and formatting the data as we've shown in preprocessing steps. You will hand off the prepped `X_train, y_train, X_test, y_test` arrays to the AutoML package, which will optimize a model and its parameters. Most arguments for AutoML have to do with the size of the desired model, the time to search for the best model, and where the model should be saved.  4) [auto-sklearn](http://automl.github.io/auto-sklearn/stable/) (Bayesian optimization)<jupyter_code>from autosklearn.regression import AutoSklearnRegressor
import sklearn.cross_validation
import sklearn.metrics

automl_r = AutoSklearnRegressor(time_left_for_this_task=60)
automl_r.fit(X_train, y_train)
y_hat = automl_r.predict(X_test)
print("R2 score", sklearn.metrics.r2_score(y_test, y_hat))<jupyter_output><empty_output><jupyter_text>Get final ensemble:<jupyter_code>print(automl_r.show_models())<jupyter_output><empty_output><jupyter_text>Get iteration scores:From docs: `(list of named tuples) Contains scores for all parameter combinations in param_grid. Each entry corresponds to one parameter setting. Each named tuple has the attributes: * parameters, a dict of parameter settings * mean_validation_score, the mean score over the cross-validation folds * cv_validation_scores, the list of scores for each fold`<jupyter_code>automl_r.grid_scores_<jupyter_output><empty_output><jupyter_text>Cross-validation results:From docs: `(dict of numpy (masked) ndarrays) A dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame. This attribute is a backward port to already support the advanced output of scikit-learn 0.18. Not all keys returned by scikit-learn are supported yet.`<jupyter_code>automl_r.cv_results_<jupyter_output><empty_output><jupyter_text>5) [TPOT](https://github.com/rhiever/tpot) (genetic algorithms) Selecting a model and parameters can be a difficult task. While you should at least conceptually understand the process and reason for your selection, a new package called `TPOT` uses genetic algorithms to help you select a model and its parameters by just feeding it the `X` and `y`. It will even write the scikit-learn code for you.<jupyter_code>from tpot import TPOTRegressor

tpot = TPOTRegressor(generations=5, population_size=20, verbosity=2)  # generations for optimization, , pop size is models
tpot.fit(X_train, y_train)
print(tpot.score(X_test, y_test))
tpot.export('tpot_boston_pipeline.py')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07968056787932565, 'fraction_numerical': 0.008873114463176575, 'mean_word_length': 3.495811727163941, 'pattern_counts': {'":': 0, '<': 89, '<?xml version=': 0, '>': 89, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 28, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '22', 'dir': 'jupyter-structured-clean-dedup', 'id': '525549', 'license': 'MIT', 'n_tokens_mistral': 3382, 'n_tokens_neox': 3236, 'n_words': 1024, 'path': 'regression.ipynb', 'repo_name': 'henchc/THW-scikit-learn', 'size': '11270'}