<jupyter_start><jupyter_text>Data normalization - normalize the data dimension that they are of approximately the same scale<jupyter_code>X_train = X_train.astype('float32') / 255
X_test = X_test.astype('float32') / 255
print('Number of train data: ' + str(len(X_train)))
print('Number of test data: ' + str(len(X_test)))<jupyter_output>Number of train data: 60000
Number of test data: 10000<jupyter_text>Split the data into train/validation/test * train set - training the model* validation set - tuning the hyperparameter and evaluate the models* test set - testing the model after the model has gone through initial vetting by the validation set<jupyter_code># break training set into train and val sets.
#(X_train, X_val) = X_train[5000:], X_train[:5000]
#(y_train, y_val) = y_train[5000:], y_train[:5000]


# generate val set-- have issues with label sizes
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=42)

print('Fashion MNIST Train - rows: ', X_train.shape[0], ' columns: ', X_train.shape[1:4])
print('Fashion MNIST Validation - rows: ', X_val.shape[0], ' columns: ', X_val.shape[1:4])
print('Fashion MNIST Test - rows: ', X_test.shape[0], ' columns: ', X_test.shape[1:4])
# reshape the input data
X_train = X_train.reshape(X_train.shape[0], 28, 28, 1)
X_val = X_val.reshape(X_val.shape[0], 28, 28, 1)
X_test = X_test.reshape(X_test.shape[0], 28, 28, 1)
 
# one-hot encode 
y_train = keras.utils.to_categorical(y_train, 10)
y_val = keras.utils.to_categorical(y_val, 10)
y_test = keras.utils.to_categorical(y_test, 10)

# print training set shape
print('X_tain shape: ', X_train.shape, 'y_train shape: ', y_train.shape)

print (X_train.shape[0], 'train set')
print (X_val.shape[0], 'validation set')
print (X_test.shape[0], 'test set')<jupyter_output>X_tain shape:  (48000, 28, 28, 1) y_train shape:  (48000, 10)
48000 train set
12000 validation set
10000 test set<jupyter_text>Model Architecture * Sequential model API in Keras* Conv2D - create a convolutional layer* Pooling() - create a pooling layer* Dropout () - apply dropout<jupyter_code>model = Sequential()

# Must define the input shape in the first layer of the neural network
model.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='relu', input_shape=(28,28,1))) 
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.3))

model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.3))

model.add(Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(MaxPooling2D(pool_size=2))
model.add(Dropout(0.1))

#model.add(Conv2D(filters=128, kernel_size=2, padding='same', activation='relu'))
#model.add(MaxPooling2D(pool_size=2))
#model.add(Dropout(0.1))


model.add(Flatten())
model.add(Dense(256, activation='relu'))
model.add(Dropout(0.3)) #0.5
model.add(Dense(10, activation='softmax'))

model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 28, 28, 64)        320       
                                                                 
 max_pooling2d (MaxPooling2D  (None, 14, 14, 64)       0         
 )                                                               
                                                                 
 dropout (Dropout)           (None, 14, 14, 64)        0         
                                                                 
 conv2d_1 (Conv2D)           (None, 14, 14, 32)        8224      
                                                                 
 max_pooling2d_1 (MaxPooling  (None, 7, 7, 32)         0         
 2D)                                                             
                                                        [...]<jupyter_text>Compile the model Configure the learning process with compile() beforing training the model.* optimizer* loss function* metrics<jupyter_code>model.compile(loss='categorical_crossentropy',
             optimizer='adam',
             metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>from keras.callbacks import ModelCheckpoint

checkpointer = ModelCheckpoint(filepath='model.weights.best.hdf5', verbose = 1, save_best_only=True)
result = model.fit(X_train,
                   y_train,
                   batch_size=64,
                   epochs=20,
                   validation_data=(X_val, y_val),
                   callbacks=[checkpointer])<jupyter_output>Epoch 1/20
750/750 [==============================] - ETA: 0s - loss: 0.7756 - accuracy: 0.7054
Epoch 00001: val_loss improved from inf to 0.48504, saving model to model.weights.best.hdf5
750/750 [==============================] - 62s 81ms/step - loss: 0.7756 - accuracy: 0.7054 - val_loss: 0.4850 - val_accuracy: 0.8204
Epoch 2/20
750/750 [==============================] - ETA: 0s - loss: 0.5101 - accuracy: 0.8075
Epoch 00002: val_loss improved from 0.48504 to 0.40229, saving model to model.weights.best.hdf5
750/750 [==============================] - 60s 79ms/step - loss: 0.5101 - accuracy: 0.8075 - val_loss: 0.4023 - val_accuracy: 0.8526
Epoch 3/20
750/750 [==============================] - ETA: 0s - loss: 0.4420 - accuracy: 0.8368
Epoch 00003: val_loss improved from 0.40229 to 0.36025, saving model to model.weights.best.hdf5
750/750 [==============================] - 60s 79ms/step - loss: 0.4420 - accuracy: 0.8368 - val_loss: 0.3603 - val_accuracy: 0.8692
Epoch 4/20
750/750 [=========[...]<jupyter_text>Load model with the bwst validation accuracy<jupyter_code># Load the weights with the best validation accuracy
model.load_weights('model.weights.best.hdf5')<jupyter_output><empty_output><jupyter_text>Test accuracy<jupyter_code># Evaluate the model on test set
score = model.evaluate(X_test, y_test, verbose=0)

# Print test accuracy
print('\n', 'Test accuracy:', score[1])
result.history['accuracy']
plt.plot(result.history['accuracy'])
result.history['val_accuracy']
plt.plot(result.history['val_accuracy'])
plt.figure(figsize=(10,4))
plt.subplot(1,2,1)

plt.plot(result.history['accuracy'], label='train accuracy', color='red')
plt.plot(result.history['val_accuracy'], label='val accuracy', color='blue')
plt.legend()

plt.subplot(1,2,2)
plt.plot(result.history['loss'], label='train loss', color='red')
plt.plot(result.history['val_loss'], label='val loss', color='blue')
plt.legend()

plt.savefig('Accuracy & Loss', dpi=150)
plt.show()
model.metrics_names<jupyter_output><empty_output><jupyter_text>Visualize prediction Now let's visualize the prediction using the model you just trained. First we get the predictions with the model from the test data. Then we print out 15 images from the test data set, and set the titles with the prediction (and the groud truth label). If the prediction matches the true label, the title will be green; otherwise it's displayed in red.<jupyter_code>labels = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
y_hat = model.predict(X_test)

# Plot a random sample of 10 test images, their predicted labels true labels
figure = plt.figure(figsize=(20, 8))
for i, index in enumerate(np.random.choice(X_test.shape[0], size=15, replace=False)):
    ax = figure.add_subplot(3, 5, i + 1, xticks=[], yticks=[])
    # Display each image
    plt.savefig('Output Sample', dpi=150)
    ax.imshow(np.squeeze(X_test[index]))
    predict_index = np.argmax(y_hat[index])
    true_index = np.argmax(y_test[index])
    # Set the title for each image
    ax.set_title("{} ({})".format(labels[predict_index], 
                                  labels[true_index]),
                                  color=("green" if predict_index == true_index else "red"))<jupyter_output><empty_output><jupyter_text>Confusion Matrix<jupyter_code>import itertools
def plot_confusion_matrix(cm, classes, normalize=False,
                         title='Confusion matrix',
                         cmap=plt.cm.Blues):
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=90)
    plt.yticks(tick_marks, classes)
    
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
    
    thresh = cm.max() / 2
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j], horizontalalignment='center', color='white' if cm[i, j] > thresh else 'black')
    
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
# predit the values from the validation dataset
y_pred = model.predict(X_test)
# convert prediction classes to one hot vectors
y_pred_classes = np.argmax(y_pred, axis=1)
# convert validation observation to one hot vectors
y_true = np.argmax(y_test, axis=1)

confusion_mtx = confusion_matrix(y_true, y_pred_classes)
plot_confusion_matrix(confusion_mtx, classes=['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 
                                              'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot'])<jupyter_output><empty_output><jupyter_text>* A large number of T-shirts/Tops are misclassified as Shirts* also lots of Shirts are misclassified as Coats  Correctly Predicted Labels<jupyter_code>correct = []
for i in range(len(y_test)):
    if (y_pred_classes[i] == y_true[i]):
        correct.append(i)
    if (len(correct) == 4):
        break
fig, ax = plt.subplots(2, 2, figsize=(12, 6))
fig.set_size_inches(10, 10)
ax[0, 0].imshow(X_test[correct[0]].reshape(28, 28), cmap='gray')
ax[0, 0].set_title('Predicted Label: ' + str(labels[y_pred_classes[correct[0]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[correct[0]]]))

ax[0, 1].imshow(X_test[correct[1]].reshape(28, 28), cmap='gray')
ax[0, 1].set_title('Predicted Label: ' + str(labels[y_pred_classes[correct[1]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[correct[1]]]))

ax[1, 0].imshow(X_test[correct[2]].reshape(28, 28), cmap='gray')
ax[1, 0].set_title('Predicted Label: ' + str(labels[y_pred_classes[correct[2]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[correct[0]]]))

ax[1, 1].imshow(X_test[correct[3]].reshape(28, 28), cmap='gray')
ax[1, 1].set_title('Predicted Label: ' + str(labels[y_pred_classes[correct[3]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[correct[3]]]))<jupyter_output><empty_output><jupyter_text>Incorrectly Predicted Classes<jupyter_code>incorrect = []
for i in range(len(y_test)):
    if (not y_pred_classes[i] == y_true[i]):
        incorrect.append(i)
    if (len(incorrect) == 4):
        break
fig, ax = plt.subplots(2, 2, figsize=(12, 6))
fig.set_size_inches(10, 10)
ax[0, 0].imshow(X_test[incorrect[0]].reshape(28, 28), cmap='gray')
ax[0, 0].set_title('Predicted Label: ' + str(labels[y_pred_classes[incorrect[0]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[incorrect[0]]]))

ax[0, 1].imshow(X_test[incorrect[1]].reshape(28, 28), cmap='gray')
ax[0, 1].set_title('Predicted Label: ' + str(labels[y_pred_classes[incorrect[1]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[incorrect[1]]]))

ax[1, 0].imshow(X_test[incorrect[2]].reshape(28, 28), cmap='gray')
ax[1, 0].set_title('Predicted Label: ' + str(labels[y_pred_classes[incorrect[2]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[incorrect[0]]]))

ax[1, 1].imshow(X_test[incorrect[3]].reshape(28, 28), cmap='gray')
ax[1, 1].set_title('Predicted Label: ' + str(labels[y_pred_classes[incorrect[3]]])
                  + '\n' + 'Acutal Label: ' + 
                  str(labels[y_true[incorrect[3]]]))<jupyter_output><empty_output><jupyter_text>Classification Report<jupyter_code>print(classification_report(y_true, y_pred_classes, target_names=labels))<jupyter_output>precision    recall  f1-score   support

 T-shirt/top       0.87      0.87      0.87      1000
     Trouser       0.99      0.98      0.98      1000
    Pullover       0.90      0.86      0.88      1000
       Dress       0.91      0.90      0.91      1000
        Coat       0.83      0.88      0.86      1000
      Sandal       0.98      0.98      0.98      1000
       Shirt       0.73      0.74      0.74      1000
     Sneaker       0.93      0.98      0.96      1000
         Bag       0.99      0.97      0.98      1000
  Ankle boot       0.98      0.94      0.96      1000

    accuracy                           0.91     10000
   macro avg       0.91      0.91      0.91     10000
weighted avg       0.91      0.91      0.91     10000
{'fraction_non_alphanumeric': 0.15499807024314935, 'fraction_numerical': 0.05642609031262061, 'mean_word_length': 3.0922299431459255, 'pattern_counts': {'":': 0, '<': 44, '<?xml version=': 0, '>': 45, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 19, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '499519', 'license': 'MIT', 'n_tokens_mistral': 4865, 'n_tokens_neox': 4348, 'n_words': 964, 'path': 'Workflow /Final Model.ipynb', 'repo_name': 'dawang09/DL__6__Project', 'size': '12955'}