<jupyter_start><jupyter_text>DAT210x - Programming with Python for DS  Module4- Lab5<jupyter_code>import pandas as pd

from scipy import misc
from mpl_toolkits.mplot3d import Axes3D
import matplotlib
import matplotlib.pyplot as plt
# Look pretty...

# matplotlib.style.use('ggplot')
plt.style.use('ggplot')<jupyter_output><empty_output><jupyter_text>Create a regular Python list (not NDArray) and name it `samples`:<jupyter_code>files = []
path = './Datasets/ALOI/32/'
import os

for filename in os.listdir(path):
    files.append(path + filename)<jupyter_output><empty_output><jupyter_text>Code up a for-loop that iterates over the images in the `Datasets/ALOI/32/` folder. Look in the folder first, so you know how the files are organized, and what file number they start from and end at.Load each `.png` file individually in your for-loop using the instructions provided in the Feature Representation reading. Once loaded, flatten the image into a single-dimensional NDArray and append it to your `samples` list.**Optional**: You can resample the image down by a factor of two if you have a slower computer. You can also scale the image from `0-255` to `0.0-1.0` if you'd like--doing so shouldn't have any effect on the algorithm's results.<jupyter_code>samples = []
for kep in files:
    img = misc.imread(kep)
    # Is the image too big? Resample it down by an order of magnitude
    img = img[::2, ::2]
    # Scale colors from (0-255) to (0-1), then reshape to 1D array per pixel, e.g. grayscale
    # If you had color images and wanted to preserve all color channels, use .reshape(-1,3)
    X = (img / 255.0).reshape(-1)
    samples.append(X)
len(samples)
type(samples)<jupyter_output><empty_output><jupyter_text>Convert `samples` to a DataFrame named `df`:<jupyter_code>df = pd.DataFrame(samples)
df.head()<jupyter_output><empty_output><jupyter_text>Import any necessary libraries to perform Isomap here, reduce `df` down to three components and using `K=6` for your neighborhood size:<jupyter_code>from sklearn import manifold

pca = manifold.Isomap(n_neighbors=6, n_components=3)
pca.fit(df)
T = pca.transform(df)<jupyter_output><empty_output><jupyter_text>Create a 2D Scatter plot to graph your manifold. You can use either `'o'` or `'.'` as your marker. Graph the first two isomap components:<jupyter_code>def Plot2D(T, title, x, y):
    # This method picks a bunch of random samples (images in your case)
    # to plot onto the chart:
    fig = plt.figure(figsize=(20,10))
    
    ax = fig.add_subplot(111)
    ax.set_title(title)
    ax.set_xlabel('Component: {0}'.format(x))
    ax.set_ylabel('Component: {0}'.format(y))
    
    # It also plots the full scatter:
    ax.scatter(T[:,x],T[:,y], marker='.',alpha=0.7)
Plot2D(T, 'izé', 0, 1)
plt.show()<jupyter_output><empty_output><jupyter_text>Chart a 3D Scatter plot to graph your manifold. You can use either `'o'` or `'.'` as your marker:<jupyter_code>def Plot3D(T, title, x, y, z):
    # This method picks a bunch of random samples (images in your case)
    # to plot onto the chart:
    fig = plt.figure(figsize=(20,10))
    
    ax = fig.add_subplot(111, projection='3d')
    ax.set_title(title)
    ax.set_xlabel('Component: {0}'.format(x))
    ax.set_ylabel('Component: {0}'.format(y))
    ax.set_zlabel('Component: {0}'.format(z))

    # It also plots the full scatter:
    ax.scatter(T[:,x],T[:,y],T[:,z], marker='.',alpha=0.7)
Plot3D(T, 'izé', 0, 1, 2)
plt.show()<jupyter_output><empty_output><jupyter_text>Answer the first two lab questions! Create another for loop. This time it should iterate over all the images in the `Datasets/ALOI/32_i` directory. Just like last time, load up each image, process them the way you did previously, and append them into your existing `samples` list:<jupyter_code>import os

path32 = './Datasets/ALOI/32/'
path32i = './Datasets/ALOI/32i/'
files32 = []
files32i = []

for filename in os.listdir(path32):
    files32.append(path32 + filename)

for filename in os.listdir(path32i):
    files32i.append(path32i + filename)
    
    
samples = []
colors = []

for kep in files32:
    img = misc.imread(kep)
    # Is the image too big? Resample it down by an order of magnitude
    img = img[::2, ::2]
    # Scale colors from (0-255) to (0-1), then reshape to 1D array per pixel, e.g. grayscale
    # If you had color images and wanted to preserve all color channels, use .reshape(-1,3)
    X = (img / 255.0).reshape(-1)
    samples.append(X)
    colors.append('b')

for kep in files32i:
    img = misc.imread(kep)
    # Is the image too big? Resample it down by an order of magnitude
    img = img[::2, ::2]
    # Scale colors from (0-255) to (0-1), then reshape to 1D array per pixel, e.g. grayscale
    # If you had color images and wanted to preserve all color channels, use .reshape(-1,3)
    X = (img / 255.0).reshape(-1)
    samples.append(X)
    colors.append('r')<jupyter_output><empty_output><jupyter_text>Convert `samples` to a DataFrame named `df`:<jupyter_code>df = pd.DataFrame(samples)
df.head()<jupyter_output><empty_output><jupyter_text>Import any necessary libraries to perform Isomap here, reduce `df` down to three components and using `K=6` for your neighborhood size:<jupyter_code>from sklearn import manifold

pca = manifold.Isomap(n_neighbors=6, n_components=3)
pca.fit(df)
T = pca.transform(df)
T.shape
len(colors)
def Plot2D(T, title, x, y, c):
    # This method picks a bunch of random samples (images in your case)
    # to plot onto the chart:
    fig = plt.figure(figsize=(20,10))
    
    ax = fig.add_subplot(111)
    ax.set_title(title)
    ax.set_xlabel('Component: {0}'.format(x))
    ax.set_ylabel('Component: {0}'.format(y))
    
    # It also plots the full scatter:
    ax.scatter(T[:,x],T[:,y], marker='.',alpha=0.7, c=c)

def Plot3D(T, title, x, y, z, c):
    # This method picks a bunch of random samples (images in your case)
    # to plot onto the chart:
    fig = plt.figure(figsize=(20,10))
    
    ax = fig.add_subplot(111, projection='3d')
    ax.set_title(title)
    ax.set_xlabel('Component: {0}'.format(x))
    ax.set_ylabel('Component: {0}'.format(y))
    ax.set_zlabel('Component: {0}'.format(z))

    # It also plots the full scatter:
    ax.scatter(T[:,x],T[:,y],T[:,z], marker='.',alpha=0.7, c=c)<jupyter_output><empty_output><jupyter_text>Create a 2D Scatter plot to graph your manifold. You can use either `'o'` or `'.'` as your marker. Graph the first two isomap components:<jupyter_code>Plot2D(T, 'izé', 0, 1, colors)
plt.show()<jupyter_output><empty_output><jupyter_text>Chart a 3D Scatter plot to graph your manifold. You can use either `'o'` or `'.'` as your marker:<jupyter_code>Plot3D(T, 'izé', 0, 1, 2, colors)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13070631970260224, 'fraction_numerical': 0.025724907063197026, 'mean_word_length': 4.262910798122066, 'pattern_counts': {'":': 0, '<': 49, '<?xml version=': 0, '>': 49, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '271354', 'license': 'MIT', 'n_tokens_mistral': 2483, 'n_tokens_neox': 2386, 'n_words': 734, 'path': 'Module4/Module4 - Lab5.ipynb', 'repo_name': 'Hazici/DAT210x', 'size': '6725'}