<jupyter_start><jupyter_text>MNIST dataset is a set of images of handwritten digits https://en.wikipedia.org/wiki/MNIST_database.The dataset has been provided to you in a .csv file. Load the MNIST dataset and split it into a training set and a test set (take the first 60,000 instances for training, and the remaining 10,000 for testing).<jupyter_code>import numpy as np
import pandas as pd
import sklearn
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
df = pd.read_csv("mnist_dataset.csv", header = 0,index_col=0)
#df = pd.read_csv("sample_data/mnist_test.csv", header = 0,index_col=0)<jupyter_output><empty_output><jupyter_text>Quick data inspection<jupyter_code>df.head(2)
df.tail(2)
df.shape
df.info()
df.describe()
# To check if we have any values and that no data was lost during its manipulation let's compute
# the sum of the values in each row

sum_rows = df.sum(axis=1)
print(sum_rows)
# Let's load the MNIST data

mnist = df
X = mnist.drop(['label'], axis=1)/ 255.0
y = mnist.label
print(X)
print(y)
feat_cols = [ 'pixel' + str(i) for i in range(X.shape[1]) ]

mnist_df = pd.DataFrame(X, columns=feat_cols)
mnist_df['y'] = y

mnist_df['label'] = mnist_df['y'].apply(lambda i: str(i))

X, y = None, None


# Generate 30 plots of randomly selected images

plt.gray()

fig= plt.figure(figsize=(16,7))

for i in range(0, 15):
    ax = fig.add_subplot(3,5, i+1, title="Digit: {}".format(str(mnist_df.loc[i, 'label'])))
    
    ax.matshow(mnist_df.loc[i, feat_cols].values.reshape((28,28)).astype(float))

plt.show()<jupyter_output><empty_output><jupyter_text>Split the data set into training and testing.<jupyter_code>#take the first 60,000 instances for training, and the remaining 10,000 for testing
df_train=df[:60000]
df_test=df[60001:]
x_train=df_train.drop(['label'], axis=1)
y_train=df_train.label
x_test=df_test.drop(['label'], axis=1)
y_test=df_test.label<jupyter_output><empty_output><jupyter_text>Train a Random Forest classifier on the dataset and time how long it takes, then evaluate the resulting model on the test set.<jupyter_code># YOUR CODE HERE
import sklearn as sk
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import make_classification
clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(x_train, y_train)
RandomForestClassifier(...)
y_train_pred=clf.predict(x_train)
print(y_train_pred)
clf = RandomForestClassifier(max_depth=2, random_state=0)
clf.fit(x_test, y_test)
RandomForestClassifier(...)
y_test_pred=clf.predict(x_test)
print(y_test_pred)<jupyter_output>[1 4 1 ... 4 2 7]<jupyter_text>Evaludation<jupyter_code>from sklearn.metrics import mean_squared_error
train_errors, test_errors = [], []
train_errors.append(mean_squared_error(y_train, y_train_pred))
test_errors.append(mean_squared_error(y_test, y_test_pred))
print(train_errors)
print(test_errors)<jupyter_output>[7.031803180318032]<jupyter_text>Next, use PCA to reduce the dataset’s dimensionality, with an explained variance ratio of 95%. Train a new Random Forest classifier on the reduced dataset and see how long it takes.<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
X2=df.drop(['label'], axis=1)
Y2=df.label
X2.columns
# build an array of features only.
minst_feature_cols = [item for item in X2.columns]


from sklearn.decomposition import PCA

pca_100f = PCA(n_components=100)

minst_100d = pca_100f.fit_transform(X2[minst_feature_cols])
pca_100f.components_
pca_100f.explained_variance_ratio_
pca_100f_cum_var = np.cumsum(pca_100f.explained_variance_ratio_)
pca_100f_cum_var
pca_200f = PCA(n_components=200)

minst_200d = pca_200f.fit_transform(X2[minst_feature_cols])
pca_200f.components_
pca_200f.explained_variance_ratio_
pca_200f_cum_var = np.cumsum(pca_200f.explained_variance_ratio_)
pca_200f_cum_var<jupyter_output><empty_output><jupyter_text>With only 200 dimension, it will acheive 96.6% variance Compare PCA to the original dataset<jupyter_code>minst_200d_df=pd.DataFrame(minst_200d)
minst_200d_df
minst_200d_df.head(2)
clf.fit(minst_200d_df, Y2)
y_200d=clf.predict(minst_200d)
print(y_200d)
reduced_errors= []
reduced_errors.append(mean_squared_error(Y2, y_200d))
print(reduced_errors)
print(test_errors)<jupyter_output>[7.031803180318032]<jupyter_text>The PCA reduced dimension dataset has a MSE of 5.09, which is lower than the original data set which has a MSE of 7.03. The performance of reduced dataset model is better.  Use t-SNE to reduce the MNIST dataset, show result graphically.<jupyter_code>import time
N=10000
mnist_df_subset = df.loc[:N, :].copy()
time_start = time.time()
# import TSNE from sklearn
from sklearn.manifold import TSNE



x_minst_subeset=mnist_df_subset.drop(['label'], axis=1)
tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300)

tsne_result = tsne.fit_transform(x_minst_subeset.values)

print('t-SNE done! Time elapsed: {} seconds'.format(time.time()-time_start))
mnist_df_subset['tsne-2d-one'] = tsne_result[:, 0]
mnist_df_subset['tsne-2d-two'] = tsne_result[:, 1]

plt.figure(figsize=(16,10))
sns.scatterplot(x='tsne-2d-one', y='tsne-2d-two', hue='label',
               palette=sns.color_palette('hls', 10),
               data=mnist_df_subset,
               alpha=0.6)<jupyter_output><empty_output><jupyter_text>Compare with other dimensionality methods: _Locally Linear Embedding_ (LLE) or _Multidimensional scaling_ (MDS). Locally Linear Embedding (LLE)<jupyter_code>from sklearn.manifold import LocallyLinearEmbedding

lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10, random_state=41)
X_reduced_sr = lle.fit_transform(x_test)
#lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10, random_state=41)
#X_reduced_sr = lle.fit_transform(X2)
plt.title("Locally Linear Embedding", fontsize=14)
plt.scatter(X_reduced_sr[:, 0], X_reduced_sr[:, 1],c=y_test,cmap=plt.cm.hot)
plt.xlabel("$z_1$", fontsize=18)
plt.ylabel("$z_2$", fontsize=18)
plt.axis([-0.015, 0.03, -0.05, 0.05])
plt.grid(True)

plt.show()<jupyter_output><empty_output><jupyter_text>Multidimensional scaling (MDS)<jupyter_code>from sklearn.manifold import MDS
X2.shape
embedding = MDS(n_components=2)
X_transformed = embedding.fit_transform(x_test[:1000])
X_transformed.shape
print(X_transformed)
#embedding = MDS(n_components=2)
#X_transformed = embedding.fit_transform(x_test[:200])
#X_transformed.shape
#(200, 2)
t=y_test[:1000]
plt.title("MDS", fontsize=14)
plt.scatter(X_transformed[:, 0], X_transformed[:, 1],c=t, cmap=plt.cm.hot)
plt.xlabel("$z_1$", fontsize=18)
plt.ylabel("$z_2$", fontsize=18)
plt.grid(True)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11923019094872951, 'fraction_numerical': 0.04450458577657495, 'mean_word_length': 7.004813477737666, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '525301', 'license': 'MIT', 'n_tokens_mistral': 2786, 'n_tokens_neox': 2517, 'n_words': 542, 'path': 'MNIST Data with dimension reduction.ipynb', 'repo_name': 'zoe-xu-1234/Data-Science-Profile', 'size': '6651'}