<jupyter_start><jupyter_text>*This notebook contains an excerpt from the book [Machine Learning for OpenCV](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv) by <NAME>.The code is released under the [MIT license](https://opensource.org/licenses/MIT),and is available on [GitHub](https://github.com/mbeyeler/opencv-machine-learning).**Note that this excerpt contains only the raw code - the book is rich with additional explanations and illustrations.If you find this content useful, please consider supporting the work by[buying the book](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-opencv)!*   Reducing the Dimensionality of the DataThe **curse of dimensionality**: The number of data points needed to fill the available space grows exponentially with the number of dimensions.In practice, the curse of dimensionality means that for a givensample size, there is a maximum number of features above which the performance of our classifier willdegrade rather than improve.That's why you often want to reduce the **dimensionality** of your data.> Here the book offers a detailed treatment of dimensionality reduction in 2D and 3D spaces, as well as intuitive explanations of PCA, ICA, and NMF.  Implementing Principal Component Analysis (PCA) in OpenCVOne of the most common dimensionality reduction techniques is called **PrincipalComponent Analysis (PCA)**.What PCA does is rotate all data points until the data lie aligned with the two axes (the twoinset vectors) that explain most of the spread of the data. PCA considers these two axes tobe the most informative, because if you walk along them, you can see most of the datapoints separated. In more technical terms, PCA aims to transform the data to a newcoordinate system by means of an orthogonal linear transformation. The new coordinatesystem is chosen such that if you project the data onto these new axes, the first coordinate(called the first principal component) observes the greatest variance.Let's have a look at some random data drawn from a multivariate Gaussian:<jupyter_code>import numpy as np
mean = [20, 20]
cov = [[5, 0], [25, 25]]
np.random.seed(42)
x, y = np.random.multivariate_normal(mean, cov, 1000).T<jupyter_output><empty_output><jupyter_text>We can plot this data using Matplotlib:<jupyter_code>import matplotlib.pyplot as plt
plt.style.use('ggplot')
%matplotlib inline
plt.figure(figsize=(10, 6))
plt.plot(x, y, 'o', zorder=1)
plt.axis([0, 40, 0, 40])
plt.xlabel('feature 1')
plt.ylabel('feature 2');<jupyter_output><empty_output><jupyter_text>In order to compute the principal components, we need to stack the `x` and `y` coordinates:<jupyter_code>X = np.vstack((x, y)).T<jupyter_output><empty_output><jupyter_text>Then we can compute PCA on the feature matrix `X`. We also specify an empty array `np.array([])` for themask argument, which tells OpenCV to use all data points in the feature matrix:<jupyter_code>import cv2
mu, eig = cv2.PCACompute(X, np.array([]))
eig<jupyter_output><empty_output><jupyter_text>Let's plot the eigenvectors of the decomposition on top of the data:<jupyter_code>plt.figure(figsize=(10, 6))
plt.plot(x, y, 'o', zorder=1)
plt.quiver(mean[0], mean[1], eig[:, 0], eig[:, 1], zorder=3, scale=0.2, units='xy')
plt.text(mean[0] + 5 * eig[0, 0], mean[1] + 5 * eig[0, 1], 'u1', zorder=5, 
         fontsize=16, bbox=dict(facecolor='white', alpha=0.6))
plt.text(mean[0] + 7 * eig[1, 0], mean[1] + 4 * eig[1, 1], 'u2', zorder=5, 
         fontsize=16, bbox=dict(facecolor='white', alpha=0.6))
plt.axis([0, 40, 0, 40])
plt.xlabel('feature 1')
plt.ylabel('feature 2');<jupyter_output><empty_output><jupyter_text>We said PCA rotates the data so that the two axes (`x` and `y`) are aligned with the first two principal components:<jupyter_code>X2 = cv2.PCAProject(X, mu, eig)<jupyter_output><empty_output><jupyter_text>If we plot this, we should see the blob of data rotated so that the most spread is along the `x` axis:<jupyter_code>plt.figure(figsize=(10, 6))
plt.plot(X2[:, 0], X2[:, 1], 'o')
plt.xlabel('first principal component')
plt.ylabel('second principal component')
plt.axis([-20, 20, -10, 10])<jupyter_output><empty_output><jupyter_text>Implementing Independent Component Analysis (ICA)Other useful dimensionality reduction techniques that are closely related to PCA areprovided by scikit-learn, but not OpenCV. We mention them here for the sake ofcompleteness. **Independent Component Analysis (ICA)** performs the same mathematicalsteps as PCA, but it chooses the components of the decomposition to be as independent aspossible from each other.In scikit-learn, ICA is available from the `decomposition` module:<jupyter_code>from sklearn import decomposition<jupyter_output><empty_output><jupyter_text>Like all other optimizers, first instantiate, then use the `fit_transform` method:<jupyter_code>ica = decomposition.FastICA()
X2 = ica.fit_transform(X)<jupyter_output>/home/ubuntu/anaconda3/envs/Python3/lib/python3.5/site-packages/sklearn/decomposition/fastica_.py:116: UserWarning: FastICA did not converge. Consider increasing tolerance or the maximum number of iterations.
  warnings.warn('FastICA did not converge. Consider increasing '<jupyter_text>We can plot the projected data on the first two independent components:<jupyter_code>plt.figure(figsize=(10, 6))
plt.plot(X2[:, 0], X2[:, 1], 'o')
plt.xlabel('first independent component')
plt.ylabel('second independent component')
plt.axis([-0.2, 0.2, -0.2, 0.2])
plt.savefig('ica.png')<jupyter_output><empty_output><jupyter_text>Implementing Nonnegative Matrix Factorization (NMF)Another useful dimensionality reduction technique is called **Nonnegative MatrixFactorization (NMF)**. It again implements the same basic mathematical operations as PCAand ICA, but it has the additional constraint that it only operates on non-negative data. Inother words, we cannot have negative values in our feature matrix if we want to use NMF;the resulting components of the decomposition will all have non-negative values as well.In scikit-learn, calling NMF works exactly like ICA:<jupyter_code>nmf = decomposition.NMF()
X2 = nmf.fit_transform(X)<jupyter_output><empty_output><jupyter_text>However, the resulting decomposition looks noticeably different from both PCA and ICA,which is a hallmark of NMF:<jupyter_code>plt.figure(figsize=(10, 6))
plt.plot(X2[:, 0], X2[:, 1], 'o')
plt.xlabel('first non-negative component')
plt.ylabel('second non-negative component')
plt.axis([0, 7, -0, 15])<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10560904783738347, 'fraction_numerical': 0.020174232003668042, 'mean_word_length': 6.1597374179431075, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 50, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '256384', 'license': 'MIT', 'n_tokens_mistral': 2129, 'n_tokens_neox': 1967, 'n_words': 730, 'path': 'notebooks/04.02-Reducing-the-Dimensionality-of-the-Data.ipynb', 'repo_name': 'qabacus/opencv-machine-learning', 'size': '6552'}