<jupyter_start><jupyter_text>In this notebook, I am going to try to build a model that reaches above 0.995 scores on kaggle's leaderboard (top 10%). To do so, we first need to pick the best CNN architecture using RandomizedSearchCV. Also we have to try data augmentation, dropout, and use a learning schedule.  Import libraries<jupyter_code>import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split, RandomizedSearchCV

from keras.models import Sequential
from keras.layers import InputLayer, Conv2D, MaxPool2D, Flatten, Dense, Dropout
from keras.optimizers import Adam
from keras.preprocessing.image import ImageDataGenerator
from keras.utils import to_categorical
from keras.wrappers.scikit_learn import KerasClassifier
from keras.callbacks import EarlyStopping
import keras<jupyter_output><empty_output><jupyter_text>Loading, preprocessing, and split the data<jupyter_code>train_set = pd.read_csv("../input/digit-recognizer/train.csv")
test_set = pd.read_csv("../input/digit-recognizer/test.csv")
X_train = train_set.drop("label", axis=1)
y_train = train_set["label"]

X_train = X_train / 255.
X_test = test_set / 255.

X_train = X_train.values.reshape(-1, 28, 28, 1)
X_test = X_test.values.reshape(-1, 28, 28, 1)

# label encoded to one hot vectors
y_train = to_categorical(y_train, num_classes=10)

X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=0.1, random_state=42)<jupyter_output><empty_output><jupyter_text>Build the model A typical CNN architecture generally stack few convolutional layers and pooling layer, then repeat the operation several times.Let's start by creating a function that build and compile our keras model.<jupyter_code>def build_model(hidden_layers=1, feature_maps=16, kernel_size=3, n_neurons=32, dropout=0.1):
    model = Sequential([])
    model.add(InputLayer(input_shape=(28, 28, 1)))
    for n in range(hidden_layers):
        model.add(Conv2D((n + 1) * feature_maps, kernel_size, activation="relu", padding="same"))
        model.add(Conv2D((n + 1) * feature_maps, kernel_size, activation="relu", padding="same"))
        model.add(MaxPool2D())
        model.add(Dropout(dropout))   
    model.add(Flatten())
    model.add(Dense(n_neurons))
    model.add(Dropout(dropout))
    model.add(Dense(10, activation="softmax"))
    model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
    return model<jupyter_output><empty_output><jupyter_text>We will use randomized search rather than grid search because there are many hyperparameters and the model may perform slightly better so it's not worth the computational cost.In order to do RandomizedSearchCV we need to wrap our keras model using a KerasClassifier class.<jupyter_code>keras.backend.clear_session()
np.random.seed(42)

keras_clf = KerasClassifier(build_model)

params = {
    "hidden_layers": [1, 2, 3],
    "feature_maps": [16, 24, 32],
    "n_neurons": [64, 128, 256],
    "dropout": [0.2, 0.3, 0.4]
}
batch_size = 64

search_cv = RandomizedSearchCV(keras_clf, params, n_iter=15, cv=3, verbose=2)
search_cv.fit(X_train, y_train, epochs=30,
              validation_data=(X_valid, y_valid),
              callbacks=EarlyStopping(patience=7),
              batch_size=batch_size,
              verbose=0)
model = search_cv.best_estimator_.model
model.summary()
search_cv.best_score_<jupyter_output><empty_output><jupyter_text>Data augmentationWe will use the Keras' `ImageDataGenerator` class to apply on-the-fly data augmentation. Note that this class only returns the randomly transformed training data.To learn more about data augmentation and the Keras' `ImageDataGenerator` class please read:* The blog post [Keras ImageDataGenerator and Data Augmentation](https://www.pyimagesearch.com/2019/07/08/keras-imagedatagenerator-and-data-augmentation/)* This stackoverflow question: https://stackoverflow.com/questions/51677788/data-augmentation-in-pytorch<jupyter_code>n_epochs = 30
s = n_epochs * len(X_train) // batch_size # number of steps in n_epochs epochs
learning_rate = keras.optimizers.schedules.ExponentialDecay(0.001, s, 0.1)

img_generator = ImageDataGenerator(
    rotation_range=0.1,
    zoom_range=0.1,
    width_shift_range=0.1, 
    height_shift_range=0.1
)

model.compile(optimizer=Adam(learning_rate), loss="categorical_crossentropy", metrics=["accuracy"])

history = model.fit(img_generator.flow(X_train, y_train, batch_size=batch_size, seed=42),
                   epochs=n_epochs, validation_data=(X_valid, y_valid),
                   callbacks=[EarlyStopping(patience=7)])<jupyter_output>Epoch 1/30
591/591 [==============================] - 11s 18ms/step - loss: 0.0913 - accuracy: 0.9725 - val_loss: 0.0241 - val_accuracy: 0.9938
Epoch 2/30
591/591 [==============================] - 11s 18ms/step - loss: 0.0582 - accuracy: 0.9829 - val_loss: 0.0241 - val_accuracy: 0.9938
Epoch 3/30
591/591 [==============================] - 11s 19ms/step - loss: 0.0522 - accuracy: 0.9849 - val_loss: 0.0235 - val_accuracy: 0.9938
Epoch 4/30
591/591 [==============================] - 11s 18ms/step - loss: 0.0516 - accuracy: 0.9847 - val_loss: 0.0176 - val_accuracy: 0.9933
Epoch 5/30
591/591 [==============================] - 11s 18ms/step - loss: 0.0450 - accuracy: 0.9865 - val_loss: 0.0203 - val_accuracy: 0.9950
Epoch 6/30
591/591 [==============================] - 12s 20ms/step - loss: 0.0429 - accuracy: 0.9865 - val_loss: 0.0212 - val_accuracy: 0.9952
Epoch 7/30
591/591 [==============================] - 11s 18ms/step - loss: 0.0401 - accuracy: 0.9882 - val_loss: 0.0166 - val_accuracy:[...]<jupyter_text>Ensemble LearningHere we are going to try to combine 8 CNN to have a better classifier. We predict the class that gets the most votes.Hopefully, this will give us a slightly better accuracy than our first model.<jupyter_code>train_set = pd.read_csv("../input/digit-recognizer/train.csv")
test_set = pd.read_csv("../input/digit-recognizer/test.csv")
X_train = train_set.drop("label", axis=1)
y_train = train_set["label"]
X_train = X_train / 255.
X_test = test_set / 255.
X_train = X_train.values.reshape(-1, 28, 28, 1)
X_test = X_test.values.reshape(-1, 28, 28, 1)
y_train = to_categorical(y_train, num_classes=10)

nets = 8
ensemble = [search_cv.best_estimator_.model for _ in range(nets)]
history = [0] * nets

for j, clf in enumerate(ensemble):
    X_train2, X_valid2, y_train2, y_valid2 = train_test_split(X_train, y_train, test_size=0.15, random_state=42)   
    clf.compile(optimizer=Adam(learning_rate), loss="categorical_crossentropy", metrics=["accuracy"])
    history[j] = clf.fit(img_generator.flow(X_train2, y_train2, batch_size=batch_size, seed=42),
                         epochs=n_epochs, validation_data=(X_valid2, y_valid2), verbose=0)
    print("CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}".format(
          j+1,n_epochs,max(history[j].history['accuracy']),max(history[j].history['val_accuracy']) ))
from scipy.stats import mode

y_pred = np.empty([nets, len(X_test)])

for clf_index, clf in enumerate(ensemble):
    y_pred[clf_index] = np.argmax(clf.predict(X_test), axis=1)

y_pred_majority_votes, n_votes = mode(y_pred, axis=0)
results = y_pred_majority_votes.reshape([-1]).astype(int)
results = pd.Series(results,name="Label")
submission = pd.concat([pd.Series(range(1,28001),name = "ImageId"),results],axis = 1)
submission.to_csv("MNIST-CNN-ENSEMBLE.csv",index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14591413668583655, 'fraction_numerical': 0.05429985288217199, 'mean_word_length': 5.104489795918368, 'pattern_counts': {'":': 4, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '26289', 'license': 'MIT', 'n_tokens_mistral': 2871, 'n_tokens_neox': 2566, 'n_words': 599, 'path': 'kaggle-mnist/digit-recognition-with-cnn-top-10%.ipynb', 'repo_name': 'Rouizi/machine-learning-portfolio', 'size': '7477'}