<jupyter_start><jupyter_text>Exploratory Data Analysis of MNIST  Table of Content  Handling Import<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import copy
from IPython.display import display
import os
import random
import cv2<jupyter_output><empty_output><jupyter_text>Importing Data<jupyter_code>data_path = '../../data/raw/mnist/'
test_name = 'test.csv'
train_name = 'train.csv'
test_df = pd.read_csv(os.path.join(data_path, test_name))
train_df = d=pd.read_csv(os.path.join(data_path, train_name))
display(train_df.shape)
display(test_df.shape)
display(train_df.head())
display(test_df.head())<jupyter_output><empty_output><jupyter_text>Reviewing data<jupyter_code>random_index_train = random.randint(0, train_df.shape[0])
random_train = np.array(train_df.iloc[random_index_train])
plt.imshow(random_train[1:].reshape((28,28)), cmap='gray')
plt.show()
random_index_test = random.randint(0, test_df.shape[0])
random_test = np.array(test_df.iloc[random_index_test])
plt.imshow(random_test.reshape((28,28)), cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Converting data into imageWe have to convert the above said data from a (784,) shaped series into a (28,28) shaped array so that we can work on it properly and perform image processing algorithms on it  Coverting FunctionThis function will convert the said series of the dataframe into a Series object containing the 28x28 image and its label.Note: Here we are returning a Series object with the column mentioned so that the resultant Dataframe on using the `apply` function will have the specified column names as its own column names.<jupyter_code>def convert_series_to_image(s):
    if s.size == 785:
        img = np.array(s[1:]).reshape((28,28))
        label = s[0]
        return pd.Series([img, label], index=['image', 'label'])
    else:
        img = np.array(s).reshape((28,28))
        return pd.Series([img], index=['image'])<jupyter_output><empty_output><jupyter_text>Performing the conversionHere we will apply the above function on our dataframe to get a ouput dataframe which has 2 columns i.e the 28x28 image and its corresponding label.Note: Here we have passed a parameter called `result_type`. This parameter decides what will be the format of the output after applying the function row-wise. Here the output will be a dataframe.  Train Data<jupyter_code>train_image_df = train_df.apply(convert_series_to_image, result_type='expand', axis='columns')
display(train_image_df.head())
display(train_image_df.shape)
random_index = random.randint(0, train_image_df.shape[0])
random_image_data = train_image_df.iloc[random_index]
plt.imshow(random_image_data[0], cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()<jupyter_output><empty_output><jupyter_text>Test Data<jupyter_code>test_image_df = test_df.apply(convert_series_to_image, result_type='expand', axis='columns')
display(test_image_df.head())
display(test_image_df.shape)
random_index = random.randint(0, test_image_df.shape[0])
random_image_data = train_image_df.iloc[random_index]
plt.imshow(random_image_data[0], cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Experimenting with Pre-Processing Methods  ThresholdingWe can see tat the figure is not quite clear, we can make it clearn using a thresholding function to convert any pixel less than 250 to 0.This way the gray areas of the image will turn into black and the image will look much clear.<jupyter_code>img_threshold = copy.deepcopy(random_image_data[0])
plt.imshow(img_threshold, cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()
img_threshold[img_threshold < 220] = 0
plt.imshow(img_threshold, cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()<jupyter_output><empty_output><jupyter_text>After some hit and trial I came to the conclusion that the threshold value of 220 works best in this scenario.  ClosingFrom thresholding we can observe that some images show broken links, as a result the image might not be able to denote properly which digit it wants to represent. This breaking of line can be filled using the concept of closing.<jupyter_code>img_close = cv2.morphologyEx(random_image_data[0].astype(np.uint8), cv2.MORPH_CLOSE, np.ones((5,5)))<jupyter_output><empty_output><jupyter_text>After some hit and trial I have come to the conclusion of using a (5,5) kernel.<jupyter_code>plt.imshow(img_close, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Closing using custom Kernel<jupyter_code>kernel = np.array([[0, 0, 1, 0, 0],
                   [0, 1, 1, 1, 0],
                   [1, 1, 1, 1, 1],
                   [0, 1, 1, 1, 0],
                   [0, 0, 1, 0, 0]], np.uint8)
img_close_custom = cv2.morphologyEx(random_image_data[0].astype(np.uint8), cv2.MORPH_CLOSE, kernel)
plt.imshow(img_close_custom, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Erosion<jupyter_code>img_erosion = copy.deepcopy(random_image_data[0])
plt.imshow(img_erosion, cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()
img_erosion = cv2.erode(img_erosion.astype(np.uint8), kernel=np.ones((2,2), np.uint8))
plt.imshow(img_erosion, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>After some hit and trial I feel a (2,2) filter is good for erosion.It feels that just using erosion is much better alternative to thresholding+closing method.  Reverse Thresholdingcoverting a pixel above a particular threshold to full value i.e. 255.<jupyter_code>img_reverse_threshold = copy.deepcopy(random_image_data[0])
plt.imshow(img_reverse_threshold, cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()
img_reverse_threshold[img_reverse_threshold >= 220] = 255
plt.imshow(img_reverse_threshold, cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()<jupyter_output><empty_output><jupyter_text>Binary TransformationHere we apply both thresholding and reverse thresholding so that the output only contains value 0 and 255.<jupyter_code>img_binary = copy.deepcopy(random_image_data[0])
plt.imshow(img_binary, cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()
def binary_transform(img, limit):
    img[img < limit] = 0
    img[img > limit] = 255
    return img
plt.imshow(binary_transform(img_binary, 200), cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Skeletonization<jupyter_code>img_skeleton = copy.deepcopy(random_image_data[0])
plt.imshow(img_skeleton, cmap='gray')
plt.title(str(random_image_data[1]))
plt.show()
def skeleton_transform(img, kernel):
    skeleton = np.zeros((28, 28), np.uint8)
    size = np.size(img)
    img = img.astype(np.uint8)
    while(True):
        #plt.subplot(1, 5, 1, title='image').imshow(img, cmap='gray')
        erode = cv2.erode(img, kernel=kernel)
        #plt.subplot(1, 5, 2, title='erode').imshow(erode, cmap='gray')
        temp = cv2.dilate(erode, kernel)
        #plt.subplot(1, 5, 3, title='dilate').imshow(temp, cmap='gray')
        temp = cv2.subtract(img, temp)
        #plt.subplot(1, 5, 4, title='subtract').imshow(temp, cmap='gray')
        skeleton = cv2.bitwise_or(skeleton, temp)
        #plt.subplot(1, 5, 5, title='skeleton').imshow(skeleton, cmap='gray')
        #plt.show()
        img = erode.copy()

        zeros = size - cv2.countNonZero(img)
        if zeros == size:
            break
    return skeleton
kernel_skeleton = np.ones((2, 2), np.uint8)
img_skeleton = skeleton_transform(img_skeleton, kernel_skeleton)
plt.imshow(img_skeleton, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>ConclusionLets see the result and try to conclude which preprocessing fits our needs the best.<jupyter_code>plt.figure(figsize=(10, 6))
plt.title('PreProcessing Methods Output')
rcols = 2
ncols = 4
plt.subplot(rcols, ncols, 1, title='Threshold').imshow(img_threshold, cmap='gray')
plt.subplot(rcols, ncols, 2, title='Closing').imshow(img_close, cmap='gray')
plt.subplot(rcols, ncols, 3, title='Closing Custom').imshow(img_close_custom, cmap='gray')
plt.subplot(rcols, ncols, 4, title='Erosion').imshow(img_erosion, cmap='gray')
plt.subplot(rcols, ncols, 5, title='Reverse Threshold').imshow(img_reverse_threshold, cmap='gray')
plt.subplot(rcols, ncols, 6, title='Binary Threshold').imshow(img_binary, cmap='gray')
plt.subplot(rcols, ncols, 7, title='Skeleton').imshow(img_skeleton, cmap='gray')<jupyter_output><empty_output><jupyter_text>Here can see that Binary Threshold and skeleton Performs the best for our use case scenario.  Applying the Pre-Process methods to the datasetNow that we have experimented with the methods on a random input, lets apply the methdo on all the test and train data.  Binary Threshold<jupyter_code>## Process Parameter
PROCESS_BINARY_LIMIT = 200
PROCESS_BINARY_TITLE = 'processed_image_binary'
def process_image_binary(s):
    img = copy.deepcopy(s['image'])
    img = binary_transform(img, PROCESS_BINARY_LIMIT)
    return pd.Series([img], index=[PROCESS_BINARY_TITLE])
train_image_df[PROCESS_BINARY_TITLE] = train_image_df.apply(process_image_binary, axis='columns')
display(train_image_df.head())
display(train_image_df.shape)
test_image_df[PROCESS_BINARY_TITLE] = test_image_df.apply(process_image_binary, axis='columns')
display(test_image_df.head())
display(test_image_df.shape)<jupyter_output><empty_output><jupyter_text>Comparing Result of Processing<jupyter_code>def plot_random_images(df, n, title):
    rows = int(np.ceil(n ** 0.5))
    cols = np.ceil(2*(n ** 0.5))
    plt.figure(figsize=(20, 10))
    j = 1
    for i in range(n):
        rand_index = random.randint(0, df.shape[0])
        img = df['image'].iloc[rand_index]
        pr_img = df[title].iloc[rand_index]
        subplt_l = plt.subplot(rows, cols, j)
        subplt_l.imshow(img, cmap='gray')
        subplot_r = plt.subplot(rows, cols, j + rows)
        subplot_r.imshow(pr_img, cmap='gray')
        if j % rows == 0:
            j = j + rows
        j += 1
    plt.show()
plot_random_images(train_image_df, 9, PROCESS_BINARY_TITLE)
plot_random_images(test_image_df, 9, PROCESS_BINARY_TITLE)<jupyter_output><empty_output><jupyter_text>Skeleton with Binary<jupyter_code>## Process Parameter
PROCESS_SKELETON_KERNEL = np.ones((2,2), np.uint8)
PROCESS_BINARY_LIMIT = 200
PROCESS_SKEL_BINARY_TITLE = 'processed_image_skeleton_binary'
def process_image_skeleton_binary(s):
    img = copy.deepcopy(s['image'])
    img = skeleton_transform(img, PROCESS_SKELETON_KERNEL)
    img = binary_transform(img, PROCESS_BINARY_LIMIT)
    return pd.Series([img], index=[PROCESS_SKEL_BINARY_TITLE])
train_image_df[PROCESS_SKEL_BINARY_TITLE] = train_image_df.apply(process_image_skeleton_binary, axis='columns')
display(train_image_df.head())
display(train_image_df.shape)
test_image_df[PROCESS_SKEL_BINARY_TITLE] = test_image_df.apply(process_image_skeleton_binary, axis='columns')
display(test_image_df.head())
display(test_image_df.shape)<jupyter_output><empty_output><jupyter_text>Comparing Result of Processing<jupyter_code>plot_random_images(train_image_df, 9, PROCESS_SKEL_BINARY_TITLE)
plot_random_images(test_image_df, 9, PROCESS_SKEL_BINARY_TITLE)<jupyter_output><empty_output><jupyter_text>Skeleton<jupyter_code>## Process Parameter
PROCESS_SKELETON_KERNEL = np.ones((2,2), np.uint8)
PROCESS_SKEL_TITLE = 'processed_image_skeleton'
def process_image_skeleton(s):
    img = copy.deepcopy(s['image'])
    img = skeleton_transform(img, PROCESS_SKELETON_KERNEL)
    return pd.Series([img], index=[PROCESS_SKEL_TITLE])
train_image_df[PROCESS_SKEL_TITLE] = train_image_df.apply(process_image_skeleton, axis='columns')
display(train_image_df.head())
display(train_image_df.shape)
test_image_df[PROCESS_SKEL_TITLE] = test_image_df.apply(process_image_skeleton, axis='columns')
display(test_image_df.head())
display(test_image_df.shape)<jupyter_output><empty_output><jupyter_text>Comparing Result of Processing<jupyter_code>plot_random_images(train_image_df, 9, PROCESS_SKEL_TITLE)
plot_random_images(test_image_df, 9, PROCESS_SKEL_TITLE)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11920805257466101, 'fraction_numerical': 0.01755261625488728, 'mean_word_length': 6.425571340333539, 'pattern_counts': {'":': 0, '<': 87, '<?xml version=': 0, '>': 87, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '21', 'dir': 'jupyter-structured-clean-dedup', 'id': '587881', 'license': 'RSA-MD', 'n_tokens_mistral': 4439, 'n_tokens_neox': 4205, 'n_words': 881, 'path': 'notebooks/exploratory_data_analysis/eda.ipynb', 'repo_name': 'SB-Jr/Hand-Digit-Recognition', 'size': '12021'}