<jupyter_start><jupyter_text>Datasets Download<jupyter_code>!wget 'http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz'
!wget 'http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz'
!gunzip train-images-idx3-ubyte.gz
!gunzip train-labels-idx1-ubyte.gz<jupyter_output>--2018-12-20 17:04:02--  http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Resolving yann.lecun.com (yann.lecun.com)... 172.16.31.10
Connecting to yann.lecun.com (yann.lecun.com)|172.16.31.10|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9912422 (9.5M) [application/x-gzip]
Saving to: ‘train-images-idx3-ubyte.gz’

train-images-idx3-u 100%[===================>]   9.45M  11.8MB/s    in 0.8s    

2018-12-20 17:04:03 (11.8 MB/s) - ‘train-images-idx3-ubyte.gz’ saved [9912422/9912422]

--2018-12-20 17:04:07--  http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Resolving yann.lecun.com (yann.lecun.com)... 172.16.31.10
Connecting to yann.lecun.com (yann.lecun.com)|172.16.31.10|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28881 (28K) [application/x-gzip]
Saving to: ‘train-labels-idx1-ubyte.gz’

train-labels-idx1-u 100%[===================>]  28.20K  --.-KB/s    in 0.08s   

2018-12-20 17:04:07 (342 KB/s) - ‘train-labels-id[...]<jupyter_text>Clone Repo<jupyter_code>!git clone https://github.com/ZeyadZanaty/image-clustering.git<jupyter_output>Cloning into 'image-clustering'...
remote: Enumerating objects: 63, done.[K
remote: Counting objects:   1% (1/63)   [K
remote: Counting objects:   3% (2/63)   [K
remote: Counting objects:   4% (3/63)   [K
remote: Counting objects:   6% (4/63)   [K
remote: Counting objects:   7% (5/63)   [K
remote: Counting objects:   9% (6/63)   [K
remote: Counting objects:  11% (7/63)   [K
remote: Counting objects:  12% (8/63)   [K
remote: Counting objects:  14% (9/63)   [K
remote: Counting objects:  15% (10/63)   [K
remote: Counting objects:  17% (11/63)   [K
remote: Counting objects:  19% (12/63)   [K
remote: Counting objects:  20% (13/63)   [K
remote: Counting objects:  22% (14/63)   [K
remote: Counting objects:  23% (15/63)   [K
remote: Counting objects:  25% (16/63)   [K
remote: Counting objects:  26% (17/63)   [K
remote: Counting objects:  28% (18/63)   [K
remote: Counting objects:  30% (19/63)   [K
remote: Counting objects:  31% (20/63)   [K
remote: Counting objects:  33% ([...]<jupyter_text>Imports<jupyter_code>import os
os.chdir('image-clustering')
from DataReader import DataReader
from KMeans import KMeans
import matplotlib.pyplot as plt
import numpy as np<jupyter_output><empty_output><jupyter_text>Data Reading<jupyter_code>data_reader = DataReader('../','mnist')
tr_data, tr_class_labels, tr_subclass_labels = data_reader.get_train_data()
data_reader.plot_imgs(tr_data,25,True)<jupyter_output><empty_output><jupyter_text>K-Means<jupyter_code>kmeans = KMeans(n_clusters=10,max_iter=200)
kmeans.fit(tr_data,tr_class_labels)<jupyter_output>60000it [00:05, 10780.14it/s]
1133it [00:00, 11327.55it/s]<jupyter_text>Mean Images<jupyter_code>data_reader.plot_imgs(kmeans.centroids,len(kmeans.centroids))<jupyter_output><empty_output><jupyter_text>Loss<jupyter_code>print('Loss')
plt.plot(range(kmeans.iterations),kmeans.loss_per_iteration)
plt.show()<jupyter_output>Loss<jupyter_text>Clustered Samples<jupyter_code>for key,data in list(kmeans.clusters['data'].items()):
    print('Cluster:',key,'Label:',kmeans.clusters_labels[key])
    data_reader.plot_imgs(data[:min(25,data.shape[0])],min(25,data.shape[0]))<jupyter_output>Cluster: 0 Label: 9<jupyter_text>Accuracy<jupyter_code>print('[cluster_label,no_occurrence_of_label,total_samples_in_cluster,cluster_accuracy]\n',kmeans.clusters_info)
print('Accuracy:',kmeans.accuracy)<jupyter_output>[cluster_label,no_occurrence_of_label,total_samples_in_cluster,cluster_accuracy]
 [[9, 2812, 6733, 0.4176444378434576], [7, 4179, 4561, 0.9162464371848279], [1, 3680, 5976, 0.6157965194109772], [4, 2613, 7154, 0.3652502096729103], [1, 3017, 5286, 0.5707529322739311], [0, 4695, 5008, 0.9375], [8, 3076, 7143, 0.43063138737225254], [6, 4940, 5704, 0.8660589060308556], [2, 4194, 4723, 0.8879949184840144], [3, 3832, 7712, 0.49688796680497926]]
Accuracy: 0.6504763715078206
{'fraction_non_alphanumeric': 0.1745583038869258, 'fraction_numerical': 0.13945818610129565, 'mean_word_length': 5.5423728813559325, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 33, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '87593', 'license': 'MIT', 'n_tokens_mistral': 2204, 'n_tokens_neox': 1807, 'n_words': 286, 'path': 'mnist_image_clustering.ipynb', 'repo_name': 'AdeelAnjum-IT/ImageClustering', 'size': '4245'}