<jupyter_start><jupyter_text>**Converting Output into one hot code**A one hot encoding is a representation of categorical variables as binary vectors. This first requires that the categorical values be mapped to integer values. Then, each integer value is represented as a binary vector that is all zero values except the index of the integer, which is marked with a 1 and as this is a multi classification problem so we can convert the output class values into one-hot format which is simply a binary matrix, i.e.value 0 will be converted to one-hot format as [1, 0, 0, 0, 0, 0, 0, 0, 0]value 1 will be converted to one-hot format as [0, 1, 0, 0, 0, 0, 0, 0, 0] etc<jupyter_code># one hot encode outputs'
y_train = np_utils.to_categorical(y_train)
num_classes = y_train.shape[1]
num_classes
# Reshaping the Image for CNN 2-dimesional input in [samples][pixels][width][height]
x_train = x_train.reshape(x_train.shape[0], 1, 28, 28).astype('float32')
x_test = x_test.reshape(x_test.shape[0], 1, 28, 28).astype('float32')
num_pixels = x_train.shape[1]
print (num_pixels, x_train.shape, x_test.shape)<jupyter_output><empty_output><jupyter_text>EnsembleUsing ensemble of cnn for training and prediction. Using 10 CNNs.Model idea and code from [here](https://www.kaggle.com/cdeotte/25-million-images-0-99757-mnist)<jupyter_code>nn = 10
model = [0]*nn

for j in range(nn):
    model[j] = Sequential()
    model[j].add(Conv2D(filters=32, kernel_size=(3, 3), input_shape=(1, 28, 28), activation='relu',data_format='channels_first'))
    model[j].add(BatchNormalization())
    model[j].add(Conv2D(filters=32, kernel_size=(3, 3), activation='relu'))
    model[j].add(BatchNormalization())
    model[j].add(Conv2D(32, kernel_size = 5, strides=2, padding='same', activation='relu'))
    model[j].add(BatchNormalization())
    model[j].add(Dropout(0.4))
    
    model[j].add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
    model[j].add(BatchNormalization())
    model[j].add(Conv2D(filters=64, kernel_size=(3, 3), activation='relu'))
    model[j].add(BatchNormalization())
    model[j].add(Conv2D(64, kernel_size = 5, strides=2, padding='same', activation='relu'))
    model[j].add(BatchNormalization())
    model[j].add(Dropout(0.4))
    
    model[j].add(Conv2D(128, kernel_size = 4, activation='relu'))
    model[j].add(BatchNormalization())
    model[j].add(Flatten())
    model[j].add(Dropout(0.4))
    model[j].add(Dense(10, activation='softmax'))
    
    # Compile model
    model[j].compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
# With data augmentation to prevent overfitting
datagen = ImageDataGenerator(
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        vertical_flip=False)  # randomly flip images


datagen.fit(x_train)<jupyter_output><empty_output><jupyter_text>Train 10 CNNSEverytime before the training, it divides the dataset into training and validation<jupyter_code># DECREASE LEARNING RATE EACH EPOCH
annealer = LearningRateScheduler(lambda x: 1e-3 * 0.95 ** x)
history = [0] * nn
epochs = 50

# Fit the model
for j in range(nn):
    x_train2, x_val, y_train2, y_val = train_test_split(x_train, y_train, test_size = 0.10, random_state=random_seed)
    history[j] = model[j].fit_generator(datagen.flow(x_train2,y_train2, batch_size=64),
                              epochs = epochs, validation_data = (x_val,y_val),
                              verbose = 0, steps_per_epoch=(len(x_train)//64),validation_steps=(len(x_val)//64),callbacks=[annealer])
    print("CNN {0:d}: Epochs={1:d}, Train accuracy={2:.5f}, Validation accuracy={3:.5f}".format(
        j+1,epochs,max(history[j].history['acc']),max(history[j].history['val_acc']) ))<jupyter_output><empty_output><jupyter_text>Ensemble Predictions and submitting the resultTill now I got 0.997 accuracy with just 5 models.<jupyter_code>results = np.zeros( (x_test.shape[0],10) )
for j in range(nn):
    results = results + model[j].predict(x_test)
results = np.argmax(results,axis = 1)
results = pd.Series(results,name="Label")
submission = pd.concat([pd.Series(range(1,28001),name = "ImageId"),results],axis = 1)
submission.to_csv("ENSEMBLE.csv",index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13743864346273985, 'fraction_numerical': 0.035698348951361, 'mean_word_length': 5.0827679782903665, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '604673', 'license': 'MIT', 'n_tokens_mistral': 1632, 'n_tokens_neox': 1514, 'n_words': 373, 'path': 'Notebooks/Ensemble_CNN.ipynb', 'repo_name': 'ankurshukla03/MNIST_Digit', 'size': '4482'}