<jupyter_start><jupyter_text>1. Statiscal learning: The setting and the estimator object 1.1 Datasets2D arrays, .shape(n_samplpes, n_features) 1.1.1 load_datasets<jupyter_code>from sklearn import datasets
iris = datasets.load_iris()
data = iris.data
data.shape, iris.feature_names, iris.target_names
digits = datasets.load_digits()
digits.images.shape
%matplotlib inline
import matplotlib.pyplot as plt

plt.imshow(digits.images[-1], cmap=plt.cm.gray_r)
# to use this dataset with the scikit, we transform each 8*8 image into a feature vector of length 64
data = digits.images.reshape((digits.images.shape[0], -1))
data.shape, digits.images<jupyter_output><empty_output><jupyter_text>1.1.2 dataset split: train dataset & test dataset<jupyter_code>from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.2, random_state=20 )<jupyter_output><empty_output><jupyter_text>1.1.3  Preprocessing* **Normalizing data**       For many estimators, including the SVMs, having datasets with unit standard deviation for each feature is important to get good prediction.    * based on the mean and std    * Scaling features to a range<jupyter_code>import sklearn.preprocessing as sk_preprocessing
# based on the mean and std
scaler = sk_preprocessing.StandardScaler().fit(iris.data)
new_x = scaler.transform(iris.data)
new_x[:5]
# normalization to a range
scaler = sk_preprocessing.MinMaxScaler(feature_range=(0, 1)).fit(iris.data)
new_x = scaler.transform(iris.data)
new_x[:4]<jupyter_output><empty_output><jupyter_text>* **regularization**       Regulazation is set by the C parameter in models: a small value for C means more regularization; a large value for C means less regularization.<jupyter_code>new_x = sk_preprocessing.normalize(iris.data, norm='l2')
new_x[:4]<jupyter_output><empty_output><jupyter_text>1.2 Estimator's objects**Fitting data**: An estimator is any object that learns from data; it may be a classification, regression or clustering algorithm or a transformer that extracts/filters useful features from raw data.```>>>estimator.fit(data)```**Estimator parameters**: All the paramaters of an estimator can be set when it instantiated or by modifying the corresponding attribute:```>>>estimator = Estimator(param1=1, param2=2)>>>eatimator.param1```**Estimated parameters**: All the estimated parameters are attributes of the estimator object ending an underscore```>>>estimator.estimated_param_```  2. Supervised learning: Predicting an output variable from high-dimensional observations* **the problem solved in supervised learning** :    Supervised learning consists in learning the link between two datasets: the observed data x and an external variable y that we are trying to predict, usually called "target" or "labels". Most often, y is a 1D array of length n_samples.        All supervised estimators in scikit-learning implements a fit(x,y) method to fit the model and a predict(x) method that, given unlabeled observations x, returns the predict labels y. * classification & regression  2.1 Nearest neighbor and the curse of dimensionality* **Classifying irises**<jupyter_code>import numpy as np
from sklearn import datasets
iris = datasets.load_iris()
iris_x = iris.data
iris_y = iris.target
np.unique(iris_y)<jupyter_output><empty_output><jupyter_text>2.1.1 K-Nearest neighbors classifier  KNN* **Training set and testing set**While experimenting with any learning algotithm, it is important not to test the prediction of an estimator on the data used to fit the estimator as this would not be evaluating the performance of the estimator on new data. This is why datasets are often split into *train* and *test* data.```sklearn.model_test.train_test_split()```<jupyter_code>## Split iris data in train and test data
# A random permutation, to split the data randomly
np.random.seed(0)
indices = np.random.permutation(len(iris_x))
iris_x_train = iris_x[indices[:-10]]
iris_y_train = iris_y[indices[:-10]]
iris_x_test = iris_x[indices[-10:]]
iris_y_test = iris_y[indices[-10:]]

# Create and fit a nearest-neighbor classifier
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier()
knn.fit(iris_x_train, iris_y_train)
knn.predict(iris_x_test), iris_y_test
knn.score(iris_x_test, iris_y_test)<jupyter_output><empty_output><jupyter_text>2.1.2 The curse of dimensionalityIf the number of features is $p$, you now require $n\sim1/d^p$. As $p$ becomes large, the number of training points required for a good estimator grows exponentially. This is called the curse of dimensionality and is a core problem that machine learning addresses.  2.2 Linear model: from regression to sparsity* **Diabetes dataset**  2.2.1 Linear regressionLinearRegression fits a linear model to the data set by adjusting a set of parameters in order to make the sum of the squared residuals of the model as small as possible.Linear model: $ y = X \beta + \varepsilon$* X: data* $y$: target variable* $\beta$: Coeffcients* $\varepsilon$: Observation noise<jupyter_code>from sklearn import linear_model
diabetes = datasets.load_diabetes()
diabetes_x_train = diabetes.data[:-20]
diabetes_y_train = diabetes.target[:-20]
diabetes_x_test = diabetes.data[-20:]
diabetes_y_test = diabetes.target[-20:]

regr = linear_model.LinearRegression()
regr.fit(diabetes_x_train, diabetes_y_train)
regr.coef_
# The mean square error
np.mean((regr.predict(diabetes_x_test)-diabetes_y_test)**2)
# Explained variance score: 1 is perfect prediction
# and 0 means that there is no linear relationship between x and y. R^2  
regr.score(diabetes_x_test, diabetes_y_test)<jupyter_output><empty_output><jupyter_text>2.2.2 ShrinkageIf there are few data points per dimension, noise in the observations induces high variance.<jupyter_code>x = np.c_[.5, 1].T
y = [5, 1]
test = np.c_[0, 2].T
regr = linear_model.LinearRegression()

import matplotlib.pyplot as plt
plt.figure()

np.random.seed(0)
for _ in range(6):
    this_x = .1*np.random.normal(size=(2, 1)) + x
    regr.fit(this_x, y)
    plt.plot(test, regr.predict(test))
    plt.scatter(this_x, y, s=3)<jupyter_output><empty_output><jupyter_text>A solution in high-dimensional statistical learning is to *shrink* the regression coefficients to zero: any two randomly chosen set of observations are likely to be uncorrelated. This is called **Ridge regression**<jupyter_code>regr = linear_model.Ridge(alpha=.1)
plt.figure()

np.random.seed(0)
for _ in range(6):
    this_x = 0.1*np.random.normal(size=(2,1)) + x
    regr.fit(this_x, y)
    plt.plot(test, regr.predict(test))
    plt.scatter(this_x, y, s=3)<jupyter_output><empty_output><jupyter_text>**bias/variance tradeoff**: the larger the ridge alpha parameter, the higher the bias and the lower the variance<jupyter_code># choose alphs to minimize left out error
alphas = np.logspace(-4, -1, 6)
alphas
from __future__ import print_function
print([regr.set_params(alpha=alpha).fit(diabetes_x_train, diabetes_y_train).score(diabetes_x_test, diabetes_y_test) for alpha in alphas])<jupyter_output>[0.5851110683883531, 0.5852073015444677, 0.5854677540698492, 0.5855512036503916, 0.5830717085554161, 0.570589994372801]<jupyter_text>**Note**: Capturing in the fitted parameters noise that prevents the model to generalize to new data is called overfitting. The bias introduced by the ridge regression is called a regulation.  2.2.3 SparsityTo improve the conditioning of the problem(i.e. mitigating the The curse of dimensionality), it would be interesting to select only the informative features and set non-informative ones. Ridge regression will decrease their contribution, but not set them to zero. Another penalization approach, called Lasso(least absolute shrinkage and selection operator), can set some coefficients to zero. Such methods are called **sparse method** and sparsity can be seen as an application of **Occam's razor**: prefer simpler models.<jupyter_code>regr = linear_model.Lasso()
scores = [regr.set_params(alpha=alpha).fit(diabetes_x_train, diabetes_y_train).score(diabetes_x_test, diabetes_y_test) for alpha in alphas]
best_alpha = alphas[scores.index(max(scores))]
regr.alpha = best_alpha
regr.fit(diabetes_x_train, diabetes_y_train)
regr.coef_<jupyter_output><empty_output><jupyter_text>* Different  algorithm for the same problemDifferent algorithm can be used to solve the same problem. For instance, the Lasso object in scikit-learn solves the lasso regression problem using a *coordinate descent* method, this is efficient on large dataset. However, the LassoLars object using the LARS algorithm is very efficient for problems in which the weight vector estimator is very sparse(i.e. problems with very few observations).  2.2.4 ClassificationFor classification, as in the labeling iris task, linear regression is not the right approach as it will give too much weight to data far from the dicision frontier. A linear approach is to fit a *sigmoid* function or *Logistic* function:$$ y = sigmoid(X\beta - offset) + \epsilon=\frac{1}{1 + exp(-X\beta + offset)} + \epsilon$$<jupyter_code>logistic = linear_model.LogisticRegression(C=1e5)
logistic.fit(iris_x_train, iris_y_train)<jupyter_output><empty_output><jupyter_text>* **Multiclass classification**If you have several classes to predict, an option often used is to fit one-versus-all classifiers and them use a voting heuristic for the final desicion.* **Shrinkage and sparsity with logistic regression**The C parameter controls the amount of regularization in the LogisticRegression object: a large value for C results in less regularization. Penalty="l2" gives Shrinkage(i.e. non-sparse coefficients), while penalty="l1" gives Sparsity.<jupyter_code>from sklearn import neighbors, linear_model
from sklearn.model_selection import train_test_split
digits_x = digits.data
digits_y = digits.target
digits_x_train, digits_x_test, digits_y_train, digits_y_test = train_test_split(digits_x, digits_y, test_size=0.1, random_state=20)
knn1 =neighbors.KNeighborsClassifier()
knn1.fit(digits_x_train, digits_y_train)
logstic1 = linear_model.LogisticRegression(C=1e5)
logstic1.fit(digits_x_train, digits_y_train)
knn1.score(digits_x_test, digits_y_test), logstic1.score(digits_x_test, digits_y_test)<jupyter_output><empty_output><jupyter_text>2.3  Support vector machines (SVMs) 2.3.1 Linear SVMsSupport Vector Machine belong to the discriminant model family: they try to find a combination of samples to build a plane maximizing the margin between the two classes. Regularization is set by the C parameter: a small value for C means the margin is calculated using many or all of the observations around the separating line(more regularization); a large value for C means the margin is calculated on observations close to the separating line(less regularization).|Unregularized SVM   |Regularized SVM (default)   ||:------|:------||| |SVM can be used in regression-SVR(Support Vector Regression), or in classification-SVC(Support Vector Classification).<jupyter_code>def make_meshgrid(x, y, h=0.02):
    """Create a mesh of points to plot in 
    Parameters
    ----------
    x: data to base x-axis meshgrid on
    y: data to base y-axis meshgrid on
    h: stepsize for meshgrid, optional
    
    Returns:
    ----------
    xx, yy: ndarray
    
    """
    x_min, x_max = x.min() - 1, x.max() + 1
    y_min, y_max = y.min() - 1, y.max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    return xx, yy
def plot_contours(ax, clf, xx, yy, **params):
    """Plot the decision boundaries for a classifier.
    
    Parameters:
    -----------
    ax: matplotlib axes object
    clf: a classifier
    xx: meshgrid ndarray
    yy: meshgrid ndarray
    params: dictionary of params to pass to contourf, optional
    """
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    out = ax.contourf(xx, yy, Z, **params)
    return out
from sklearn import svm
X = iris.data[:, :2]
y = iris.target
C = 1.0
# We do not scale our data since we want to plot the support vectors
models = (svm.SVC(kernel='linear', C=C), svm.LinearSVC(C=C), svm.SVC(kernel='rbf', gamma=0.7, C=C), svm.SVC(kernel='poly', degree=3, C=C))
models = (clf.fit(X, y) for clf in models)

# title for the plots
titles = ('SVM with linar kernel', 'LinearSVC (linear kernel)', 'SVC with RBF kernel', 'SVC with polynomial(degree 3) kernel')
# Set-up 2*2 grid for plotting
fig, sub = plt.subplots(2,2)
plt.subplots_adjust(wspace=0.4, hspace=0.4)

X0, X1 = X[:, 0], X[:,1]
xx, yy = make_meshgrid(X0, X1)

for clf, title, ax in zip(models, titles, sub.flatten()):
    plot_contours(ax, clf, xx, yy, cmap=plt.cm.coolwarm, alpha=0.8)
    ax.scatter(X0, X1, c=y, cmap=plt.cm.coolwarm, s=20, edgecolors='k')
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xlabel('Sepal length')
    ax.set_ylabel('Sepal length')
    ax.set_xticks(())
    ax.set_yticks(())
    ax.set_title(title)
    
plt.show()<jupyter_output><empty_output><jupyter_text>2.3.2 Using kernels    Classes are not always linearly separable in feature space. The solution is to build a decision function that is not linear but may be polynomial instead.    | Linear kernel | Polynomial kernel | RBF kernel(Radial Basis Function) |-|:-:|:-:||  |  |  ||>>>svc = svm.SVC(kernel='linear')|>>>svc=svm.SVC(kernel='poly',degree=3)>>>degree:polynomial degree|>>>svc=svm.SVC(kernel='rbf')>>>gamma:inverse of size of radial kernel|  3. Model selection: choosing estimators and their parameters 3.1 Score, and cross-validated scoresScore--Bigger is better.<jupyter_code>from sklearn import datasets, svm
import numpy as np
digits = datasets.load_digits()
X_digits = digits.data
y_digits = digits.target
svc = svm.SVC(C=1, kernel='linear')
svc.fit(X_digits[:-100],y_digits[:-100]).score(X_digits[-100:], y_digits[-100:])
x_folds = np.array_split(X_digits, 3)
y_folds = np.array_split(y_digits, 3)
x_folds[0].shape, X_digits.shape
scores = []
for k in range(3):
    X_train = list(x_folds)
    x_test = X_train.pop(k)
    x_train = np.concatenate(X_train)
    y_train = list(y_folds)
    y_test = y_train.pop(k)
    y_train = np.concatenate(y_train)
    scores.append(svc.fit(x_train, y_train).score(x_test, y_test))
scores<jupyter_output><empty_output><jupyter_text>3.2 Cross-validation generatorsA collection of classes can be used to generate lists of train/test indices for popular cross-validation strategies.   *KFold().split* method & *cross_val_score*<jupyter_code>from sklearn.model_selection import KFold, cross_val_score
X = ['a', 'a', 'b', 'c', 'c', 'c']
k_fold = KFold(n_splits=3)
for train_indices, test_indices in k_fold.split(X):
    print("Train: %s | test: %s" %(train_indices, test_indices))
[ svc.fit(digits_x[train], digits_y[train]).score(digits_x[test], digits_y[test]) for train,test in k_fold.split(digits_x)]
cross_val_score(svc, digits_x, digits_y, cv=k_fold, n_jobs=1)
# specify an alternative scoring method
cross_val_score(svc, digits_x, digits_y, cv=k_fold, scoring='precision_macro')<jupyter_output><empty_output><jupyter_text>Cross-validation generators* **KFold(n_splits, shuffle, random_state)**: splits it into K folds, trains on K-1 and then tests on the left-out.* **StratifiedKFold(n_splits, shuffle, random_state)**:  Same as K-Fold but preserves the class distribution within each fold.* **GroupKFold(n_splits)**: Ensure that the same group is not in both testing and training sets.* **ShuffleSplit(n_splits, test_size, train_size, random_state)**: Generates train/test indices based on random permutaion.* **StratifiedShuffleSplit**: Same as shuffle split but preserves the class distribution within each iteration.* **GroupShuffleSplit**: Ensures that the same group is not in both testing and training sets.* **LeaveOneGroupOut()**: Takes a group array to group observations.* **LeavePGroupOut(n_groups)**: Leave P groups out.* **LeaveOneOut()**: Leave one observation out.* **LeavePOut(P)**: Leave P observations out.* **PredefinedSplit**: Generates train/test indices based on predefined splits.<jupyter_code># generates train/test indices based on random permutation
from sklearn.model_selection import ShuffleSplit
ss = ShuffleSplit(n_splits=3, test_size=0.5)
for train,test in ss.split(X):
    print(train,test)
## exercise
svc = svm.SVC(kernel='linear')
C_s = np.logspace(-10, 0, 10)

scores = []
scores_std = []

for c in C_s:
    svc.C = c
    score = cross_val_score(svc, digits_x, digits_y, cv=ss, n_jobs=1)
    scores.append(np.mean(score))
    scores_std.append(np.std(score))
    
scores
plt.figure(1, figsize=(4, 3))
plt.clf()
plt.semilogx(C_s, scores)
plt.semilogx(C_s, np.array(scores) + np.array(scores_std), 'b--')
plt.semilogx(C_s, np.array(scores) - np.array(scores_std), 'b--')
locs, labels = plt.yticks()
plt.yticks(locs, list(map(lambda x: "%g" %x, locs)))
plt.ylabel("CV scores")
plt.xlabel("Parmeter C")
plt.ylim(0, 1.1)
plt.show()
locs<jupyter_output><empty_output><jupyter_text>3.3 Grid-search and cross-validation estimators 3.3.1 Grid-searchscikit-learn provides an object that, given data, computes the score during the fit of an estimator on a parameter grid and chooses the parameters to maximize the cross-validation score. This object takes an estimator during hte construction and exposes an estimator API:<jupyter_code>from sklearn.model_selection import GridSearchCV, cross_val_score
Cs = np.logspace(-6, -1, 10)
clf = GridSearchCV(estimator=svc, param_grid=dict(C=Cs), n_jobs=-1)
clf.fit(digits_x[:1000], digits_y[:1000])
clf.best_score_
clf.best_estimator_.C
clf.score(digits_x[1000:], digits_y[1000:])
cross_val_score(clf, digits_x, digits_y)<jupyter_output><empty_output><jupyter_text>By default, the GridSearchCV uses a 3-fold cross-validation. However, if it detects that a classifier is passed, rather than a regressor, it uses a stratified 3-fold.Two cross-validation loops are performed in parallel: one by the **GridSearchCV** estimator to set *gamma* and the other one by **cross_val_score** to measure the prediction performance of the estimator. The resulting scores are unbiased estimates of the prediction score on new data.  3.3.2 Cross-validated estimatorsCross-validation to set a parameter can be done more efficiently on an algorithm-by-algorithm basis. This is why, for certain estimators, scikit-learn exposes **Cross-validation: evaluating estimator performance** estimators that set their parameter automatically by cross-validation:<jupyter_code>from sklearn import linear_model, datasets
lasso = linear_model.LassoCV()
diabetes = datasets.load_diabetes()
X_diabetes = diabetes.data
y_diabetes = diabetes.target
lasso.fit(X_diabetes, y_diabetes)
lasso.alpha_
from sklearn.linear_model import LassoCV, Lasso
from sklearn.model_selection import KFold, GridSearchCV
# 3-fold kfold split the dataset
k_fold = KFold(n_splits=3)
clf1 = LassoCV(cv=k_fold)
clf1.fit(X_diabetes, y_diabetes)
clf1.score(X_diabetes[200:], y_diabetes[200:])
cross_val_score(clf1, X_diabetes, y_diabetes)
clf1.cv
lasso = Lasso()
alphas = np.logspace(-6,-1,10)
clf2 = GridSearchCV(estimator=lasso, param_grid=dict(alpha=alphas),n_jobs=1)
clf2.fit(digits_x[:1000], digits_y[:1000])
clf2.cv_results_
clf2.best_score_
alphas = np.logspace(-4,-0.5,30)
lasso = Lasso(random_state=0)
n_folds = 3
clf1 = GridSearchCV(estimator=lasso, param_grid=dict(alpha=alphas), cv=n_folds, refit=False)
clf1.fit(X_diabetes[:150], y_diabetes[:150])
scores = clf1.cv_results_['mean_test_score']
scores_std = clf1.cv_results_['std_test_score']

plt.figure().set_size_inches(8,6)
plt.semilogx(alphas, scores)
std_error = scores_std/ np.sqrt(n_folds)
plt.semilogx(alphas, scores+std_error, 'b--')
plt.semilogx(alphas, scores-std_error, 'r--')
plt.fill_between(alphas, scores+std_error, scores-std_error, alpha=0.3)
plt.xlabel('Alpha')
plt.ylabel('Score +/- std_error')
plt.axhline(np.max(scores), linestyle='--', color='.5')
#plt.axvline(x=0.01, linestyle='--', color='0.5')
plt.xlim([alphas[0], alphas[-1]])

## LassoCV
lasso_cv = LassoCV(alphas=alphas, random_state=0)
k_fold = KFold(3)

for k,(train, test) in enumerate(k_fold.split(digits_x, digits_y)):
    lasso_cv.fit(digits_x[train], digits_y[train])
    print('Fold {0}, cv-score: {1:0.5f}, cv-alpha: {2:0.5f}'.format(k,lasso_cv.alpha_, lasso_cv.score(digits_x[test], digits_y[test])))<jupyter_output>Fold 0, cv-score: 0.07880, cv-alpha: 0.51473
Fold 1, cv-score: 0.13738, cv-alpha: 0.56136
Fold 2, cv-score: 0.18139, cv-alpha: 0.44910<jupyter_text>4. Unsupervised learning: seeking representations of the data 4.1 Clustering: grouping observations together* ! **The problem solved in clustering**Given the iris dataset, if we knew that there were 3 types of iris, but did not have access to a taxonomist to label them: we could try a **clustering task**: split the observations into well-separated group called *clusters*.  4.1.1 K-means clusteringthe simplest clustering algorithm<jupyter_code>from sklearn import cluster, datasets
from mpl_toolkits.mplot3d import Axes3D

iris = datasets.load_iris()
X_iris = iris.data
y_iris = iris.target

k_means = cluster.KMeans(n_clusters=3)
k_means.fit(X_iris)
print(k_means.labels_[::10])
labels = k_means.labels_
fig = plt.figure(figsize=(4,3))
ax = Axes3D(fig, rect=[0,0,0.95,1], elev=48, azim=134)
ax.scatter(X_iris[:,3], X_iris[:,0], X_iris[:,2], c=labels.astype(np.float), edgecolor='k')
ax.w_xaxis.set_ticklabels([])
ax.w_yaxis.set_ticklabels([])
ax.w_zaxis.set_ticklabels([])
ax.set_xlabel('Petal width')
ax.set_ylabel('Sepal length')
ax.set_zlabel('Petal length')
ax.set_title("k_means_iris_3")
ax.dist = 12
import scipy as sp
try:
    face = sp.face(gray=True)
except AttributeError:
    from scipy import misc
    face = misc.face(gray=True)

X = face.reshape((-1,1))    

k_means = cluster.KMeans(n_clusters=5, n_init=1)
k_means.fit(X)
values = k_means.cluster_centers_.squeeze()
labels = k_means.labels_
face_compressed = np.choose(labels, values)
face_compressed.shape = face.shape

vmin = face.min()
vmax = face.max()

plt.figure(1, figsize=(3, 2.2))
plt.imshow(face, cmap=plt.cm.gray, vmin=vmin, vmax=256)<jupyter_output><empty_output><jupyter_text>4.1.2 Hierarchical agglomerative clustering: WardA **Hierarchical clustering** method is a type of cluster analysis that aims to build a hierarchy of clusters. In general, the various approaches of this technique are either:* **Agglomerative**: --bottom-up approaches: each observation starts in its own cluster, and clusters are iteratively merged in such a way to minimize a linkage criterion. This approach is particularly interesting when the clusters of interest are mode of only a few observations. When the number of clusters is large, it is much more computationally efficient than k-means.* **Divisive**: --top-down approaches: all observations start in one cluster, which is iteratively split as one moves down the hierarchy. For estimating large numbers of clusters, this approach is both slow (due to all observations starting as one cluster, which it splits recursively) and statistically ill-posed.  Connectivity-constrained clusteringWith agglomerative clustering, it is possible to specify which samples can be clustered together by giving a connectivity graph. Graphs in the scikit are represented by their adjacency matrix. Often, a sparse matrix is used. This can be useful, for instrance, to retrieve connected regions (sometimes also referred to as connected components) when clustering an image:<jupyter_code>import matplotlib.pyplot as plt

from sklearn.feature_extraction.image import grid_to_graph
from sklearn.cluster import AgglomerativeClustering
# Generate data
try:  # Scipy >= 0.16 have face in misc
    from scipy.misc import face
    face = face(gray=True)
except ImportError:
    face = sp.face(gray=True)
    
# Resize it to 10% of the original size to speed up the processing
face = sp.misc.imresize(face, 0.10) / 255
X = np.reshape(face, (-1, 1))

# Define the structure A of the data. Pixels connected to their neighbors
connectivity = grid_to_graph(*face.shape)<jupyter_output>/usr/local/lib/python2.7/dist-packages/ipykernel_launcher.py:9: DeprecationWarning: `imresize` is deprecated!
`imresize` is deprecated in SciPy 1.0.0, and will be removed in 1.2.0.
Use ``skimage.transform.resize`` instead.
  if __name__ == '__main__':<jupyter_text>Feature agglomerationWe have seem that sparsity could be used to mitigate the curse of dimensionality, i.e. an insufficient amount of observations compared to the number of features. Another approach is to merge together similar features: **feature agglomeration**. This approach can be implemented by clustering in the feature direction, in other words clustering th transposed data.<jupyter_code>images = digits.images
X = np.reshape(images, (len(images), -1))
connectivity = grid_to_graph(*images[0].shape)

agglo = cluster.FeatureAgglomeration(connectivity=connectivity, n_clusters=32)
agglo.fit(X)
X_reduced = agglo.transform(X)
X_approx = agglo.inverse_transform(X_reduced)
images_approx = np.reshape(X_approx, images.shape)
#images_approx
X_approx = agglo.inverse_transform(X_reduced)
images_approx = np.reshape(X_approx, images.shape)<jupyter_output><empty_output><jupyter_text>transform and inverse_transform methodssome estimators expose a *transform* method, for instance to reduce the dimensionality of the dataset. 4.2 Decompositions: from a signal to componects and loadings* **Componects and loadings**  If X is our multivariate data, then the problem that we are trying to solve is to rewrite it on a different observational basis: we want to learn loadings L and a set of components C such that X = L C. Different criteria exist to choose the components.  4.2.1 Principal component analysis: PCAPrincipal component analysis(PCA) selects the successive components that explain the maximum variance in the signal. The point cloud spanned by the observations above is very flat in one direction: one of the three univeriate features can almost be exactly computed using the other two. PCA finds the directions in which the data is not *flat*.When used to *transform* data, PCA can reduce the dimensionality of the data by projecting on a principal subspace.<jupyter_code># Create a signal with only a useful dimensions
x1 = np.random.normal(size=100)
x2 = np.random.normal(size=100)
x3 = x1 + x2
X = np.c_[x1, x2, x3]

from sklearn import decomposition
pca = decomposition.PCA()
pca.fit(X)
pca.explained_variance_
# As we can see, only the 2 first componects are useful
pca.n_components = 2
X_reduced = pca.fit_transform(X)
X_reduced.shape<jupyter_output><empty_output><jupyter_text>4.2.2 Independent Component Analysis: ICAIndependent component analysis(ICA) selects components so that the distribution of their loadings carries a maximum amount of independent information. It is able to recover **non-Gaussian** independent signals:<jupyter_code># Generate sample data
from scipy import signal
time = np.linspace(0, 10, 2000)
s1 = np.sin(2 * time)
np.random.seed(0)
n_samples = 2000
time = np.linspace(0, 8, n_samples)
s1 = np.sin(2*time)
s2 = np.sign(np.sin(3*time))
s3 = signal.sawtooth(2*np.pi*time) # Signal 3: saw tooth signal
S = np.c_[s1, s2, s3]
S += 0.2 * np.random.normal(size=S.shape) # Add noise
S /= S.std(axis=0) # Standardize data
# Mix data
A = np.array([[1, 1, 1], [0.5, 2, 1],[1.5, 1, 2]])  # Mixing matrix
X = np.dot(S, A.T)  # Generate observations

# Compute ICA
ica = decomposition.FastICA(n_components=3)
S_ = ica.fit_transform(X)  # Reconstruct signals
A_ = ica.mixing_           # get estimated mixing matrix

# We can prove that the ICA model applies by reverting the unmixing
np.allclose(X, np.dot(S_, A_.T) + ica.mean_)

# For comparison, compute PCA
pca = decomposition.PCA(n_components=3)
H = pca.fit_transform(X)  # Reconstruct signals based on orthogonal components
# plot results
plt.figure()
models = [X, S, S_, H]
names = ['Observations(mixed signal)', 'True Sources', 'ICA recovered signals', 'PCA recovered signals']
colors = ['red', 'steelblue', 'orange']

for ii, (model, name) in enumerate(zip(models, names), 1):
    plt.subplot(4,1,ii)
    plt.title(name)
    for sig, color in zip(model.T, colors):
        plt.plot(sig, color=color)
    
    plt.subplots_adjust(0.09, 0.04, 0.94, 0.94, 0.26, 0.46)
    plt.show()<jupyter_output><empty_output><jupyter_text>5 Putting it all together 5.1 PipeliningWe have seen that some estimators can transform data and that some estimators can predict variables. We can also create combined estimators:<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

from sklearn import linear_model, decomposition, datasets
from sklearn.pipeline import Pipeline
from sklearn.model_selection import GridSearchCV
logistic = linear_model.LogisticRegression()
pipe = Pipeline(steps=[('pca', pca), ('logistic', logistic)])

X_digits = digits.data
Y_digits = digits.target

# plot the PCA spectrum
pca.fit(X_digits)

plt.figure(1, figsize=(4, 3))
plt.clf()
plt.axes([0.2, 0.2, 0.7, 0.7])
plt.plot(pca.explained_variance_, linewidth=2)
plt.axis('tight')
plt.xlabel('n_componects')
plt.ylabel('explained_variance_')

# Prediction
n_components = [20, 40, 64]
Cs = np.logspace(-4, 4, 3)

# parameters of pipelines can be set using '__' separated parameter names:
estimator = GridSearchCV(pipe, dict(pca__n_components=n_components, logistic__C=Cs))
estimator.fit(X_digits, y_digits)
plt.axvline(estimator.best_estimator_.named_steps['pca'].n_components, linestyle=':', label='n_components chosen')
plt.legend(prop=dict(size=12))
plt.show()<jupyter_output><empty_output><jupyter_text>5.2 Face recognition with eigenfacesThe dataset used in this example is a preprocessed except of the 'Labeled Faces in the Wild', also known as LFW:<jupyter_code>"""
=======================================
Faces recognition example using eigenfaces and SVMs
=======================================
"""
from __future__ import print_function
from time import time
import logging
import matplotlib.pyplot as plt

from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.datasets import fetch_lfw_people
from sklearn.metrics import classification_report, confusion_matrix
from sklearn.decomposition import PCA
from sklearn.svm import SVC

print(__doc__)
# Display progress logs on stdout
logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')

## Download the data, if not already on disk and load it as numpy arrays
lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)

# introspect the images arrays to find the shapes (for plotting)
n_samples, h, w = lfw_people.images.shape

# for machinee learning we use the 2 data directly(as relative pixel positions info is ignored by this model)
X = lfw_people.data
n_features = X.shape[1]

# the label to predict is the id of the person
y = lfw_people.target
target_names = lfw_people.target_names
n_classes = target_names.shape[0]

print('Total dataset size:')
print('n_samples: %d' % n_samples)
print('n_features: %d' % n_features)
print('n_classes: %d' % n_classes)
# Split into a training set and a test set using a stratified k fold
# split into a training and testing set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

# Compute a PCA(eigenfaces) on the face dataset(treated as unlabeled dataset):
# unsupervised feature extraction / dimensionality reduction
n_components = 150
print('Extracing the top %d eigenfaces from %d faces' %(n_components, X_train.shape[0]))
t0 = time()
pca = PCA(n_components=n_components, svd_solver='randomized', whiten=True).fit(X_train)
print('Done in %0.3fs' %(time()-t0))

eigenfaces = pca.components_.reshape((n_components, h, w))

print('Projecting the input data on the eigenfaces orthonormal basis')
t0 = time()
X_train_pca = pca.transform(X_train)
X_test_pca = pca.transform(X_test)
print('Done in %0.3f' %(time()-t0))
## Train a SVM classification model
print('Fitting the classifier to the training set')
t0 = time()
param_grid = {'C': [1e3, 5e3, 1e4, 5e4, 1e5], 'gamma':[0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1]}
clf = GridSearchCV(SVC(kernel='rbf', class_weight='balanced'), param_grid)
clf = clf.fit(X_train_pca, y_train)
print('Done in %0.3fs' %(time()-t0))
print('Best estimator found by grid search:')
print(clf.best_estimator_)

# Quantitative evaluation of the model quality on the test set
print("Predicting people's names on the test set")
t0 = time()
y_pred = clf.predict(X_test_pca)
print('Done in %0.3fs' %(time()-t0))
print(classification_report(y_test, y_pred, target_names=target_names))
print(confusion_matrix(y_test, y_pred, labels=range(n_classes)))

# Qualitative evaluation of the prediction using matplotlib
def plot_gallery(images, titles, h, w, n_row=3, n_col=4):
    """Helper function to plot a gallery of portraits"""
    plt.figure(figsize=(1.8 * n_col, 2.4 * n_row))
    plt.subplots_adjust(bottom=0, left=0.01, right=0.99, top=0.99, hspace=0.35)
    for i in range(n_row * n_col):
        plt.subplot(n_row, n_col, i+1)
        plt.imshow(images[i].reshape((h, w)), cmap=plt.cm.gray)
        plt.title(titles[i], size=12)
        plt.xticks(())
        plt.yticks(())
        
# plot the result of the prediction on a portion of the best set
def title(y_pred, y_test, target_names, i):
    pred_name = target_names[y_pred[i]].rsplit(' ', 1)[-1]
    true_name = target_names[y_test[i]].rsplit(' ', 1)[-1]
    return 'Predicted: %s \nTrue:   %s' %(pred_name, true_name)

prediction_titles = [title(y_pred, y_test, target_names, i) for i in range(y_pred.shape[0]) ]
plot_gallery(X_test, prediction_titles, h, w)

# plot the gallery of the most significative eigenfaces
eigenface_titles = ['eigenface %d' % i for i in range(eigenfaces.shape[0])]
plot_gallery(eigenfaces, eigenface_titles, h, w)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11170181345619942, 'fraction_numerical': 0.021442495126705652, 'mean_word_length': 5.5886359213854835, 'pattern_counts': {'":': 0, '<': 102, '<?xml version=': 0, '>': 130, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '129081', 'license': 'MIT', 'n_tokens_mistral': 11949, 'n_tokens_neox': 11203, 'n_words': 3473, 'path': 'sklearn/tutorials/sklearn_1.scientific_data_processing.ipynb', 'repo_name': 'Li-Michael/learn', 'size': '33858'}