<jupyter_start><jupyter_text>function to reshape datasets for fitting in CNN model<jupyter_code>def reshape(dataset):
    temp=[]
    for i in range(len(dataset)):
        temp.append(dataset[i].reshape(28,28))
    newset=np.array(temp)
    return newset<jupyter_output><empty_output><jupyter_text>testing our model on validation sets created out from trainset itself<jupyter_code>x_train28=reshape(x)
x_train28=x_train28/255.#use this for final results
x_train28 = np.expand_dims(x_train28, -1)
x_train, x_test, y_train, y_test = train_test_split(x_train28, y, random_state=0, test_size=0.2)
cnn=Sequential()
cnn.add(Conv2D(filters=16, kernel_size=3, activation='relu', input_shape=[28, 28, 1]))
cnn.add(MaxPool2D(pool_size=2, strides=2))
cnn.add(Conv2D(filters=32, kernel_size=3, activation='relu'))
cnn.add(MaxPool2D(pool_size=2, strides=2))
cnn.add(Flatten())
cnn.add(Dropout(0.2))
cnn.add(Dense(128, activation='relu'))
cnn.add(Dropout(0.1))
cnn.add(Dense(10, activation='softmax'))
data_generator = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1, horizontal_flip=True)
train_generator = data_generator.flow(x_train, y_train, 32)
steps_per_epoch = x_train.shape[0] // 32
cnn.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model_cnn = cnn.fit(train_generator, validation_data=(x_test, y_test), steps_per_epoch=steps_per_epoch, epochs=16)
plt.plot(model_cnn.history['loss'], label='loss')
plt.plot(model_cnn.history['val_loss'], label='val_loss')
plt.legend()
results_temp=cnn.predict(x_test)
pred_digits = np.argmax(results_temp, axis=1)
results_temp=list(pred_digits)<jupyter_output><empty_output><jupyter_text>Fitting the tested model on actual test set(make sure we don't have the actual labels)<jupyter_code>cnn.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
cnn.fit(x_train28, y, epochs=20)
data_generator = ImageDataGenerator(width_shift_range=0.1, height_shift_range=0.1, zoom_range=0.1, horizontal_flip=True)
train_generator = data_generator.flow(x_train28, y, 32)
steps_per_epoch = x_train28.shape[0] // 32
cnn.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
cnn.fit(train_generator, steps_per_epoch=steps_per_epoch, epochs=45)<jupyter_output>Epoch 1/45
1312/1312 [==============================] - 11s 9ms/step - loss: 0.6358 - accuracy: 0.7878
Epoch 2/45
1312/1312 [==============================] - 11s 9ms/step - loss: 0.2710 - accuracy: 0.9139
Epoch 3/45
1312/1312 [==============================] - 12s 9ms/step - loss: 0.2025 - accuracy: 0.9377
Epoch 4/45
1312/1312 [==============================] - 12s 9ms/step - loss: 0.1673 - accuracy: 0.9469
Epoch 5/45
1312/1312 [==============================] - 12s 9ms/step - loss: 0.1500 - accuracy: 0.9533
Epoch 6/45
1312/1312 [==============================] - 12s 9ms/step - loss: 0.1350 - accuracy: 0.9574
Epoch 7/45
1312/1312 [==============================] - 12s 9ms/step - loss: 0.1239 - accuracy: 0.9617
Epoch 8/45
1312/1312 [==============================] - 12s 9ms/step - loss: 0.1199 - accuracy: 0.9618
Epoch 9/45
1312/1312 [==============================] - 14s 10ms/step - loss: 0.1131 - accuracy: 0.9651
Epoch 10/45
1312/1312 [==============================] - 12s 9ms/step - [...]<jupyter_text>testing on the testset<jupyter_code>test=reshape(testset)
test=test/255.#use this for final results
test= np.expand_dims(test, -1)

results=cnn.predict(test)
pred_digits = np.argmax(results, axis=1)
results=list(pred_digits)
output = pd.DataFrame({'ImageId': np.arange(1,len(results)+1), 'Label': results})
output.to_csv('./MNIST_submission.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.21733261339092871, 'fraction_numerical': 0.08720302375809935, 'mean_word_length': 7.107221006564552, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '344008', 'license': 'Apache-2.0', 'n_tokens_mistral': 1592, 'n_tokens_neox': 1375, 'n_words': 242, 'path': 'Code CNN.ipynb', 'repo_name': 'Gituhin/MNIST-Digits-recognizer-using-CNN', 'size': '3704'}