<jupyter_start><jupyter_text>SciPy 2016 Scikit-learn Tutorial  Manifold LearningOne weakness of PCA is that it cannot detect non-linear features.  A setof algorithms known as *Manifold Learning* have been developed to addressthis deficiency.  A canonical dataset used in Manifold learning is the*S-curve*:<jupyter_code>from sklearn.datasets import make_s_curve
X, y = make_s_curve(n_samples=1000)

from mpl_toolkits.mplot3d import Axes3D
ax = plt.axes(projection='3d')

ax.scatter3D(X[:, 0], X[:, 1], X[:, 2], c=y)
ax.view_init(10, -60);<jupyter_output><empty_output><jupyter_text>This is a 2-dimensional dataset embedded in three dimensions, but it is embeddedin such a way that PCA cannot discover the underlying data orientation:<jupyter_code>from sklearn.decomposition import PCA
X_pca = PCA(n_components=2).fit_transform(X)
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y);<jupyter_output><empty_output><jupyter_text>Manifold learning algorithms, however, available in the ``sklearn.manifold``submodule, are able to recover the underlying 2-dimensional manifold:<jupyter_code>from sklearn.manifold import Isomap

iso = Isomap(n_neighbors=15, n_components=2)
X_iso = iso.fit_transform(X)
plt.scatter(X_iso[:, 0], X_iso[:, 1], c=y);<jupyter_output><empty_output><jupyter_text>Manifold learning on the digits data We can apply manifold learning techniques to much higher dimensional datasets, for example the digits data that we saw before:<jupyter_code>from sklearn.datasets import load_digits
digits = load_digits()

fig, axes = plt.subplots(2, 5, figsize=(10, 5),
                         subplot_kw={'xticks':(), 'yticks': ()})
for ax, img in zip(axes.ravel(), digits.images):
    ax.imshow(img, interpolation="none", cmap="gray")<jupyter_output><empty_output><jupyter_text>We can visualize the dataset using a linear technique, such as PCA. We saw this already provides some intuition about the data:<jupyter_code># build a PCA model
pca = PCA(n_components=2)
pca.fit(digits.data)
# transform the digits data onto the first two principal components
digits_pca = pca.transform(digits.data)
colors = ["#476A2A", "#7851B8", "#BD3430", "#4A2D4E", "#875525",
          "#A83683", "#4E655E", "#853541", "#3A3120","#535D8E"]
plt.figure(figsize=(10, 10))
plt.xlim(digits_pca[:, 0].min(), digits_pca[:, 0].max() + 1)
plt.ylim(digits_pca[:, 1].min(), digits_pca[:, 1].max() + 1)
for i in range(len(digits.data)):
    # actually plot the digits as text instead of using scatter
    plt.text(digits_pca[i, 0], digits_pca[i, 1], str(digits.target[i]),
             color = colors[digits.target[i]],
             fontdict={'weight': 'bold', 'size': 9})
plt.xlabel("first principal component")
plt.ylabel("second principal component");<jupyter_output><empty_output><jupyter_text>Using a more powerful, nonlinear techinque can provide much better visualizations, though.Here, we are using the t-SNE manifold learning method:<jupyter_code>from sklearn.manifold import TSNE
tsne = TSNE(random_state=42)
# use fit_transform instead of fit, as TSNE has no transform method:
digits_tsne = tsne.fit_transform(digits.data)
plt.figure(figsize=(10, 10))
plt.xlim(digits_tsne[:, 0].min(), digits_tsne[:, 0].max() + 1)
plt.ylim(digits_tsne[:, 1].min(), digits_tsne[:, 1].max() + 1)
for i in range(len(digits.data)):
    # actually plot the digits as text instead of using scatter
    plt.text(digits_tsne[i, 0], digits_tsne[i, 1], str(digits.target[i]),
             color = colors[digits.target[i]],
             fontdict={'weight': 'bold', 'size': 9})<jupyter_output><empty_output><jupyter_text>t-SNE has a somewhat longer runtime that other manifold learning algorithms, but the result is quite striking. Keep in mind that this algorithm is purely unsupervised, and does not know about the class labels. Still it is able to separate the classes very well (though the classes four, one and nine have been split into multiple groups).  ExercisesCompare the results of applying isomap to the digits dataset to the results of PCA and t-SNE. Which result do you think looks best?<jupyter_code># %load solutions/22A_isomap_digits.py
from sklearn.manifold import Isomap
iso = Isomap(n_components=2)
digits_isomap = iso.fit_transform(digits.data)

plt.figure(figsize=(10, 10))
plt.xlim(digits_isomap[:, 0].min(), digits_isomap[:, 0].max() + 1)
plt.ylim(digits_isomap[:, 1].min(), digits_isomap[:, 1].max() + 1)
for i in range(len(digits.data)):
    # actually plot the digits as text instead of using scatter
    plt.text(digits_isomap[i, 0], digits_isomap[i, 1], str(digits.target[i]),
             color = colors[digits.target[i]],
             fontdict={'weight': 'bold', 'size': 9})<jupyter_output><empty_output><jupyter_text>Given how well t-SNE separated the classes, one might be tempted to use this processing for classification. Try training a K-nearest neighbor classifier on digits data transformed with t-SNE, and compare to the accuracy on using the dataset without any transformation.<jupyter_code># %load solutions/22B_tsne_classification.py
from sklearn.manifold import TSNE
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(digits.data, digits.target, random_state=1)

clf = KNeighborsClassifier()
clf.fit(X_train, y_train)
print('KNeighborsClassifier accuracy without t-SNE: {}'.format(clf.score(X_test, y_test)))

tsne = TSNE(random_state=42)
digits_tsne_train = tsne.fit_transform(X_train)
digits_tsne_test = tsne.fit_transform(X_test)

clf = KNeighborsClassifier()
clf.fit(digits_tsne_train, y_train)
print('KNeighborsClassifier accuracy with t-SNE: {}'.format(clf.score(digits_tsne_test, y_test)))<jupyter_output>KNeighborsClassifier accuracy without t-SNE: 0.9933333333333333
KNeighborsClassifier accuracy with t-SNE: 0.14444444444444443
{'fraction_non_alphanumeric': 0.11891706648389308, 'fraction_numerical': 0.028272789581905416, 'mean_word_length': 5.842907385697538, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '38711', 'license': 'MIT', 'n_tokens_mistral': 2141, 'n_tokens_neox': 1971, 'n_words': 559, 'path': 'scipy-2016-sklearn/notebooks/22 Unsupervised learning - Non-linear dimensionality reduction.ipynb', 'repo_name': 'sunny2309/scipy_conf_notebooks', 'size': '5836'}