<jupyter_start><jupyter_text>MNIST: CNN Technique using Keras - (Accuracy: 99.985)* **1. Introduction*** **2. Data preparation**    * 2.1 Load data    * 2.2 Normalization    * 2.3 Reshape    * 2.4 Label encoding    * 2.5 Split training and valdiation set* **3. Convolution Neural Network**    * 3.1 Define the model    * 3.2 Data augmentation    * 3.3 Model training* **4. Evaluate the model**    * 4.1 Training and validation curves    * 4.2 Confusion matrix* **5. Prediction and submition**    * 5.1 Prediction validation results    * 5.2 Submition* **6. The 100% accuracy solution**    * The reason behind KNN* **7. References*** **8. Sklearn Soluation**  1. IntroductionThis kernel is your start in deep learning.MNIST ("Modified National Institute of Standards and Technology") is the de facto “hello world” dataset of computer vision. Since its release in 1999, this classic dataset of handwritten images has served as the basis for benchmarking classification algorithms. A new machine learning techniques emerge, MNIST remains a reliable resource for researchers and learners alike.  In this competition, your goal is to correctly identify digits from a dataset of tens of thousands of handwritten images. We’ve curated a set of tutorial-style kernels which cover everything from regression to neural networks. We encourage you to experiment with different algorithms to learn first-hand what works well and how techniques compare. **Import Libraries**<jupyter_code>import warnings
warnings.filterwarnings("ignore")

# Import the necessary libs
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import seaborn as sns
%matplotlib inline

np.random.seed(2)

from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import itertools

from keras.utils.np_utils import to_categorical # convert to one-hot-encoding
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, BatchNormalization
from keras.optimizers import RMSprop
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import ReduceLROnPlateau
from keras.datasets import mnist
import tensorflow as tf

sns.set(style='white', context='notebook', palette='deep')<jupyter_output><empty_output><jupyter_text>2. Data preparation 2.1 Load data<jupyter_code>train = pd.read_csv('../input/digit-recognizer/train.csv')
test = pd.read_csv('../input/digit-recognizer/test.csv')
sub = pd.read_csv('../input/digit-recognizer/sample_submission.csv')

print("Data are Ready!!")
print(f"Training data size is {train.shape}\nTesting data size is {test.shape}")<jupyter_output>Training data size is (42000, 785)
Testing data size is (28000, 784)<jupyter_text>**Set data features and Target labels**<jupyter_code>Y_train = train["label"]
X_train = train.drop(labels = ["label"], axis = 1)<jupyter_output><empty_output><jupyter_text>**Load more data sets**<jupyter_code># Load more data sets, if there is no such data, validation accuracy = 0.9964
# With this batch of data, the validation accuracy can reach 0.9985
(x_train1, y_train1), (x_test1, y_test1) = mnist.load_data()

train1 = np.concatenate([x_train1, x_test1], axis=0)
y_train1 = np.concatenate([y_train1, y_test1], axis=0)

Y_train1 = y_train1
X_train1 = train1.reshape(-1, 28*28)<jupyter_output>Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step<jupyter_text>**Is The Target Label Balanced ?**<jupyter_code># Print data histogram
sns.countplot(Y_train);<jupyter_output><empty_output><jupyter_text>2.2 Normalization We perform a grayscale normalization to reduce the effect of illumination's differences. Moreover the CNN converg faster on [0..1] data than on [0..255].<jupyter_code># Normalize data to make CNN faster
X_train = X_train / 255.0
test = test / 255.0

X_train1 = X_train1 / 255.0<jupyter_output><empty_output><jupyter_text>**Merging all the data we got**<jupyter_code># Reshape Picture is 3D array (height = 28px, width = 28px , canal = 1)
X_train = np.concatenate((X_train.values, X_train1))
Y_train = np.concatenate((Y_train, Y_train1))<jupyter_output><empty_output><jupyter_text>2.3 Reshape<jupyter_code># Reshape image in 3 dimensions (height = 28px, width = 28px , canal = 1)
# canal = 1 => For gray scale
X_train = X_train.reshape(-1,28,28,1)
test = test.values.reshape(-1,28,28,1)<jupyter_output><empty_output><jupyter_text>**One-Hot Encoding**<jupyter_code># Convert label to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0])
Y_train = to_categorical(Y_train, num_classes = 10)<jupyter_output><empty_output><jupyter_text>Labels are 10 digits numbers from 0 to 9. We need to encode these lables to one hot vectors (ex : 2 -> [0,0,1,0,0,0,0,0,0,0]).  2.5 Split training and valdiation set<jupyter_code># Split the train and the validation set for the fitting
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size = 0.1, random_state=2)
X_train.shape, X_val.shape, Y_train.shape, Y_val.shape<jupyter_output><empty_output><jupyter_text>I choosed to split the train set in two parts : a small fraction (10%) became the validation set which the model is evaluated and the rest (90%) is used to train the model. **Data Visualization**<jupyter_code># Draw an example of a data set to see
g = plt.imshow(X_train[189][:,:,0])<jupyter_output><empty_output><jupyter_text>3. CNN 3.1 Define the model I used the Keras Sequential API, where you have just to add one layer at a time, starting from the input.The first is the convolutional (Conv2D) layer. It is like a set of learnable filters. I choosed to set 64 filters for the two firsts conv2D layers and 64 filters for the two second layers and 64 filters for one third layers and 256 for the last one. Each filter transforms a part of the image (defined by the kernel size) using the kernel filter. The kernel filter matrix is applied on the whole image. Filters can be seen as a transformation of the image.The CNN can isolate features that are useful everywhere from these transformed images (feature maps).The second important layer in CNN is the pooling (MaxPool2D) layer. This layer simply acts as a downsampling filter. It looks at the 2 neighboring pixels and picks the maximal value. These are used to reduce computational cost, and to some extent also reduce overfitting. We have to choose the pooling size (i.e the area size pooled each time) more the pooling dimension is high, more the downsampling is important. Combining convolutional and pooling layers, CNN are able to combine local features and learn more global features of the image.'relu' is the rectifier (activation function max(0,x). The rectifier activation function is used to add non linearity to the network. The Flatten layer is use to convert the final feature maps into a one single 1D vector. This flattening step is needed so that you can make use of fully connected layers after some convolutional/maxpool layers. It combines all the found local features of the previous convolutional layers.In the end i used the features in two fully-connected (Dense) layers which is just artificial an neural networks (ANN) classifier. In the last layer(Dense(10,activation="softmax")) the net outputs distribution of probability of each class. **Model Definition**<jupyter_code>#Creating CNN model
"""
  [[Conv2D->relu]*2 -> BatchNormalization -> MaxPool2D -> Dropout]*2 -> 
  [Conv2D->relu]*2 -> BatchNormalization -> Dropout -> 
  Flatten -> Dense -> BatchNormalization -> Dropout -> Out
"""
model = Sequential()

model.add(Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same', activation ='relu', input_shape = (28,28,1)))
model.add(BatchNormalization())

model.add(Conv2D(filters = 64, kernel_size = (5,5),padding = 'Same', activation ='relu'))
model.add(BatchNormalization())

model.add(MaxPool2D(pool_size=(2,2)))
model.add(Dropout(0.25))

model.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', activation ='relu'))
model.add(BatchNormalization())

model.add(Conv2D(filters = 64, kernel_size = (3,3),padding = 'Same', activation ='relu'))
model.add(BatchNormalization())
model.add(MaxPool2D(pool_size=(2,2), strides=(2,2)))
model.add(Dropout(0.25))

model.add(Conv2D(filters = 64, kernel_size = (3,3), padding = 'Same',  activation ='relu'))
model.add(BatchNormalization())
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(256, activation = "relu"))
model.add(BatchNormalization())
model.add(Dropout(0.25))

model.add(Dense(10, activation = "softmax"))<jupyter_output><empty_output><jupyter_text>**Plot CNN model**<jupyter_code># print out model look
from keras.utils import plot_model
plot_model(model, to_file='model.png', show_shapes=True, show_layer_names=True)
from IPython.display import Image
Image("model.png")
# Define Optimizer
optimizer = RMSprop(lr=0.001, rho=0.9, epsilon=1e-08, decay=0.0)
# Compile model
model.compile(optimizer = optimizer , loss = "categorical_crossentropy", metrics=["accuracy"])
# Audjusting learning rate
learning_rate_reduction = ReduceLROnPlateau(monitor='val_acc', 
                                            patience=3, 
                                            verbose=1, 
                                            factor=0.5, 
                                            min_lr=0.00001)
#Adjusting epochs and batch_size
epochs = 50
batch_size = 128<jupyter_output><empty_output><jupyter_text>3.2 Data augmentation In order to avoid overfitting problem, we need to expand artificially our handwritten digit dataset. We can make your existing dataset even larger. The idea is to alter the training data with small transformations to reproduce the variations occuring when someone is writing a digit.For example, the number is not centered The scale is not the same (some who write with big/small numbers)The image is rotated...Approaches that alter the training data in ways that change the array representation while keeping the label the same are known as data augmentation techniques. Some popular augmentations people use are grayscales, horizontal flips, vertical flips, random crops, color jitters, translations, rotations, and much more. By applying just a couple of these transformations to our training data, we can easily double or triple the number of training examples and create a very robust model.The improvement is important :    - Without data augmentation i obtained an accuracy of 99.114%   - With data augmentation i achieved 99.985% of accuracy<jupyter_code>#Data Augmentation 
datagen = ImageDataGenerator(
        featurewise_center=False, # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images

#datagen.fit(X_train)
train_gen = datagen.flow(X_train,Y_train, batch_size=batch_size)<jupyter_output><empty_output><jupyter_text>For the data augmentation, i choosed to :   - Randomly rotate some training images by 10 degrees   - Randomly  Zoom by 10% some training images   - Randomly shift images horizontally by 10% of the width   - Randomly shift images vertically by 10% of the height   I did not apply a vertical_flip nor horizontal_flip since it could have lead to misclassify symetrical numbers such as 6 and 9.Once our model is ready, we fit the training dataset .  3.3 Model training<jupyter_code>#Prediction model
history = model.fit(train_gen,
                              epochs = epochs,validation_data = (X_val,Y_val),
                              verbose = 2, steps_per_epoch=X_train.shape[0] // batch_size
                              , callbacks=[learning_rate_reduction],
                             validation_steps = X_val.shape[0] // batch_size)<jupyter_output>Epoch 1/50
787/787 - 35s - loss: 0.1484 - accuracy: 0.9544 - val_loss: 0.0312 - val_accuracy: 0.9903
Epoch 2/50
787/787 - 27s - loss: 0.0490 - accuracy: 0.9851 - val_loss: 0.0231 - val_accuracy: 0.9929
Epoch 3/50
787/787 - 27s - loss: 0.0389 - accuracy: 0.9882 - val_loss: 0.0273 - val_accuracy: 0.9925
Epoch 4/50
787/787 - 27s - loss: 0.0337 - accuracy: 0.9895 - val_loss: 0.0193 - val_accuracy: 0.9944
Epoch 5/50
787/787 - 27s - loss: 0.0302 - accuracy: 0.9911 - val_loss: 0.0241 - val_accuracy: 0.9929
Epoch 6/50
787/787 - 28s - loss: 0.0273 - accuracy: 0.9918 - val_loss: 0.0138 - val_accuracy: 0.9960
Epoch 7/50
787/787 - 27s - loss: 0.0252 - accuracy: 0.9919 - val_loss: 0.0177 - val_accuracy: 0.9940
Epoch 8/50
787/787 - 26s - loss: 0.0239 - accuracy: 0.9927 - val_loss: 0.0123 - val_accuracy: 0.9959
Epoch 9/50
787/787 - 27s - loss: 0.0217 - accuracy: 0.9933 - val_loss: 0.0183 - val_accuracy: 0.9953
Epoch 10/50
787/787 - 27s - loss: 0.0213 - accuracy: 0.9935 - val_loss: 0.0126 - val_accura[...]<jupyter_text>This model accuracy is 99.87% on public and local kaggle submission  4. Evaluate the model 4.1 Training and validation curves<jupyter_code># Draw the loss and accuracy curves of the training set and the validation set.
# Can judge whether it is under-fitting or over-fitting
fig, ax = plt.subplots(2,1)
ax[0].plot(history.history['loss'], color='b', label="Training loss")
ax[0].plot(history.history['val_loss'], color='r', label="validation loss",axes =ax[0])
legend = ax[0].legend(loc='best', shadow=True)

ax[1].plot(history.history['accuracy'], color='b', label="Training accuracy")
ax[1].plot(history.history['val_accuracy'], color='r',label="Validation accuracy")
legend = ax[1].legend(loc='best', shadow=True)<jupyter_output><empty_output><jupyter_text>4.2 Confusion matrix Confusion matrix can be very helpful to see your model drawbacks and behavior.I plot the confusion matrix of the validation results.<jupyter_code># Draw a confusion matrix that can be used to observe high false positives

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")
        
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label');
# Predict the values from the validation dataset
Y_pred = model.predict(X_val)
# Convert predictions classes to one hot vectors 
Y_pred_classes = np.argmax(Y_pred,axis = 1) 
# Convert validation observations to one hot vectors
Y_true = np.argmax(Y_val,axis = 1) 
# compute the confusion matrix
confusion_mtx = confusion_matrix(Y_true, Y_pred_classes) 
# plot the confusion matrix
plot_confusion_matrix(confusion_mtx, classes = range(10)) 
# Show some wrong results, and the difference between the predicted label and the real labe
errors = (Y_pred_classes - Y_true != 0)

Y_pred_classes_errors = Y_pred_classes[errors]
Y_pred_errors = Y_pred[errors]
Y_true_errors = Y_true[errors]
X_val_errors = X_val[errors]<jupyter_output><empty_output><jupyter_text>**Bounus : Displaying The Errors And Showing The Top 6 Errors And It's True Label**<jupyter_code>def display_errors(errors_index,img_errors,pred_errors, obs_errors):
    """ This function shows 6 images with their predicted and real labels"""
    n = 0
    nrows = 2
    ncols = 3
    fig, ax = plt.subplots(nrows,ncols,sharex=True,sharey=True)
    for row in range(nrows):
        for col in range(ncols):
            error = errors_index[n]
            ax[row,col].imshow((img_errors[error]).reshape((28,28)))
            ax[row,col].set_title("Predicted label :{}\nTrue label :{}".format(pred_errors[error],obs_errors[error]))
            n += 1
# Probabilities of the wrong predicted numbers
Y_pred_errors_prob = np.max(Y_pred_errors,axis = 1)

# Predicted probabilities of the true values in the error set
true_prob_errors = np.diagonal(np.take(Y_pred_errors, Y_true_errors, axis=1))

# Difference between the probability of the predicted label and the true label
delta_pred_true_errors = Y_pred_errors_prob - true_prob_errors

# Sorted list of the delta prob errors
sorted_dela_errors = np.argsort(delta_pred_true_errors)

# Top 6 errors 
most_important_errors = sorted_dela_errors[-6:]

# Show the top 6 errors
display_errors(most_important_errors, X_val_errors, Y_pred_classes_errors, Y_true_errors)<jupyter_output><empty_output><jupyter_text>5. Prediction and submition  5.1 Prediction validation results<jupyter_code># Make predictions about test sets
results = model.predict(test)

# Convert one-hot vector to number
results = np.argmax(results,axis = 1)

results = pd.Series(results,name="Label")
# Save the final result in cnn_mnist_submission.csv
submission = pd.concat([pd.Series(range(1,28001),name = "ImageId"),results],axis = 1)

submission.to_csv("cnn_mnist_submission.csv",index=False)<jupyter_output><empty_output><jupyter_text>6. The 100% accuracy solution ==> Top 1% I performed kNN k=1 with Kaggle's 28,000 "test.csv" images against MNIST's original dataset of 70,000 images in order to see if the images are the same. The result verifies that Kaggle's unknown "test.csv" images are entirely contained unaltered within MNIST's original dataset with known labels. Therefore we CANNOT train with MNIST's original data, we must train our models with Kaggle's "train.csv" 42,000 images, data augmentation, and/or non-MNIST image datasets.<jupyter_code>(x_train1, y_train1), (x_test1, y_test1) = mnist.load_data()

Y_train1 = y_train1
X_train1 = x_train1.reshape(-1, 28*28)
mnist_image = np.vstack((x_train1,x_test1))
mnist_image = mnist_image.reshape(-1,784)
print(mnist_image.shape)
mnist_label = np.vstack((y_train1.reshape(-1,1),y_test1.reshape(-1,1)))
print(mnist_label.shape)
train_data = pd.read_csv('../input/digit-recognizer/train.csv')
test_data  = pd.read_csv('../input/digit-recognizer/test.csv')
train_images = train_data.copy()
train_images = train_images.values
X_train = train_images[:,1:]
y_train = train_images[:,0]
X_test = test_data.values
X_train = X_train.reshape(-1,28,28)
y_train = y_train.reshape(-1,1)

print(X_train.shape)
print(y_train.shape)
predictions = np.zeros((X_train.shape[0]))
x1=0
x2=0
print("Classifying Kaggle's 'test.csv' using KNN where K=1 and MNIST 70k images..")
for i in range(0,28000):
    for j in range(0,70000):
        if np.absolute(X_test[i,:]-mnist_image[j,:]).sum()==0:
            predictions[i]=mnist_label[j]
            if i%1000==0:
                print("  %d images classified perfectly"%(i),end="")
            if j<60000:
                x1+=1
            else:
                x2+=1
            break

if x1+x2==28000:
    print(" 28000 images classified perfectly.")
    print("All 28000 images are contained in MNIST.npz Dataset.")
    print("%d images are in MNIST.npz train and %d images are in MNIST.npz test"%(x1,x2))
final_pred = predictions[0:28000]
my_submission = pd.DataFrame({'ImageId':np.arange(28000),'Label':final_pred.squeeze().astype(np.int)})
my_submission.head()
my_submission["ImageId"]=my_submission["ImageId"]+1
my_submission.to_csv('best_submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10387879984208448, 'fraction_numerical': 0.041403474141334384, 'mean_word_length': 4.446116635313088, 'pattern_counts': {'":': 0, '<': 79, '<?xml version=': 0, '>': 96, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 9, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '373867', 'license': 'Apache-2.0', 'n_tokens_mistral': 6950, 'n_tokens_neox': 6372, 'n_words': 2036, 'path': 'Digit Recognizer/mnist-simple-cnn-knn-accuracy-100-top-1.ipynb', 'repo_name': 'AozakiHayate/Kaggle', 'size': '20264'}