<jupyter_start><jupyter_text>The Curse of DimensionalitySupervised learning is, in essense, about *generalization*.  We fit models to training data, and we expect those models to make accuratepredictions on unseen data.  This is only possible if our trainingdata is, in some sense, similar to our testing data.  This raises someimportant questions:1) What exactly do we mean by similarity (or dissimilarity)?2) How much data do we need?  If we hope to make an accurateprediction about a testing point, we clearly need enough training datato be confident that we have seen at least one example that is similarto it.  How much is that?We'll get back to question 1).  Question 2) is related to the*dimensionality* of our data set. Question*   Imagine we have divided our data space into grid cells.  We are willing to generalize from old data as long as we have at least *one* training point in the same grid cell as our testing point.  Our granularity will be 10 cells per dimension.  So if we are making predictions from 1d data, our input space is divided up like this:    ![1d grid](images/1d_grid.png)    If we are making predictions from 2d data, our input space is divided up like this:    ![2d grid](images/2d_grid.png)    How much data do we need for one-dimensional data?  How much for two-dimensional data?  How much for 100-dimensional data? --- Another Way Of Looking At The ProblemThe cell below generates some random data points and plots a histogram of the distances between them. Try changing the dimensionality of the data and looking at how this changes the distance distribution.<jupyter_code>%matplotlib qt
import numpy as np
import matplotlib.pyplot as plt

from sklearn.metrics.pairwise import euclidean_distances

samples = 1000
dims = 2

points = np.random.rand(samples, dims)
dist = euclidean_distances(points) # get distances between all points

upper_tri = dist[np.triu_indices(samples,k=1)]
upper_tri_mu = np.mean(upper_tri)
upper_tri_std = np.std(upper_tri)

n, bins, patches = plt.hist(x=upper_tri, bins='auto', color='blue',
                            alpha=0.7)
plt.grid(axis='y', alpha=0.75)
plt.xlim(0, max(bins))
plt.xlabel('Distance',fontsize=18)
plt.ylabel('Count',fontsize=18)
plt.title(str(samples) + ' Points in ' + str(dims) +
          ' dim space ' + r'$\mu$=%0.1f std=%0.1f'%
          (upper_tri_mu,upper_tri_std),fontsize=20)
plt.show()<jupyter_output><empty_output><jupyter_text>Question* How does the number of randomly generated points that are near each other change as the dimensionality increases? --- There is Hope!High-dimensional data is a fundamental challenge in machine learning.  For example, consider image data.  Each data item is composed of many thousands of individual attributes representing the color values for each pixel.  How can we hope to work with this sort of data given the issues illustrated above???  Source of Hope \ 1 - Irrelevant (or less relevant) AttributesSometimes, not all of the attributes in the dataset are actually relevent to the learning problem we are trying to solve.  For example, our goal for the data set below is to predict the value of the final column using the other five attributes.  Take a look at this subset of the data to see if you can see the relationship:<jupyter_code>from mpl_toolkits.mplot3d import Axes3D
data = np.loadtxt('data/dims.csv', delimiter=',')
print(data[0:10, :])<jupyter_output>[[-0.12433  0.67385  1.      -0.36333  0.89254 -0.24483]
 [-0.14731  0.58515  0.      -1.00506  0.71352 -0.58812]
 [-0.99205  0.93073  0.       0.95289  0.32335  0.88688]
 [-0.59885  0.77542  1.      -2.2056   0.17221 -1.71027]
 [-1.81108  0.96133  0.5     -1.71702  0.59198 -1.65063]
 [-1.14077  0.19882  0.       0.78885  0.4011   0.15684]
 [ 0.98275  0.73843  0.       0.14252  0.71794  0.10524]
 [-0.5651   0.32222  0.       1.79837  0.52331  0.57947]
 [ 0.5575   0.96172  1.       0.41839  0.97731  0.40238]
 [ 1.30389  0.89171  0.5     -0.21014  0.71333 -0.18738]]<jupyter_text>Assuming it isn't jumping out at you, we can do some exploratory data analysys by plotting our target value against each of the different attributes.  See if you can figure out if any of the attributes appear relevant.  If you find two relevant attributes, you can try plotting a 3d scatter plot.<jupyter_code># Plot the target value against attribute 0:
plt.plot(data[:, 0], data[:, 5], '*')
plt.show()

# Uncomment the code below to create a 3d scatter plot...

# Plot the target value against attribute 0:

#fig = plt.figure()
#ax = fig.add_subplot(111, projection='3d')
#ax.scatter(data[:,0], data[:, 1], data[:, 5])
#plt.show()<jupyter_output><empty_output><jupyter_text>Question*   Which attributes are most relevant?*   What what output value would you predict for the attributes    [ 0.719  0.304  0.          0.328  0.124].  Source of Hope \ 2 - Lower Dimensional EmbeddingsSometimes,  what *appears* to be high-dimensional data is actually low-dimensional data embedded in a high dimensional space. Try creating a scatter plot of the following three-dimensional data set.<jupyter_code>data = np.loadtxt('data/embed.csv', delimiter=',')
print(data[0:10, :])<jupyter_output>[[1.04376 0.75309 0.89138]
 [0.82159 0.99307 0.20919]
 [0.50953 0.29039 0.53016]
 [0.61913 0.20959 0.82045]
 [0.51758 0.4721  0.32064]
 [0.74791 0.86306 0.24082]
 [0.86459 0.39199 1.02356]
 [0.86223 0.93845 0.34716]
 [0.79643 0.62851 0.61387]
 [1.01863 0.87677 0.69545]]<jupyter_text>Question* How would you describe the "true" dimensionality of this data?  Source of Hope \ 3 - ManifoldsSometimes, what appears to be high-demensional data is actually a low-dimensional *manifold* embedded in a high-dimensional space.  Inspect the data below as a 3d scatter plot.<jupyter_code>data = np.loadtxt('data/manifold.csv', delimiter=',')
print(data[0:10, :])<jupyter_output>[[ 0.79218  0.71804  0.64258]
 [ 0.40085  0.65681  0.23334]
 [ 0.11685  0.50176  0.55479]
 [ 0.64464  0.77364  0.34144]
 [ 1.19689  0.67484  0.71785]
 [ 0.08704  0.23069 -0.06788]
 [-0.01118  0.63169 -0.05429]
 [ 0.24954  0.59989  0.39266]
 [ 0.12     0.51133  0.4722 ]
 [ 1.2029   0.70008  0.80371]]
{'fraction_non_alphanumeric': 0.09877750611246944, 'fraction_numerical': 0.12127139364303179, 'mean_word_length': 3.7236335642802154, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '168134', 'license': 'MIT', 'n_tokens_mistral': 2454, 'n_tokens_neox': 2035, 'n_words': 765, 'path': 'dimensionality.ipynb', 'repo_name': 'spragunr/jmu_ml_dimensionality_lab', 'size': '6135'}