<jupyter_start><jupyter_text>MNIST Classification with MLP -- Built-in training/eval loops --In this notebook, I describe how to implement MLP using tf.keras and use tf.keras's build-in training/eval loops.  Also, we use only the classes that have been pre-prepared in tf.keras<jupyter_code>import tensorflow as tf
import time
import numpy as np
import sklearn
from sklearn.model_selection import train_test_split
import matplotlib
import matplotlib.pyplot as plt
import os
%matplotlib inline

print("tensorflow version: ", tf.__version__)
print("numpy version: ", np.__version__)
print("scikit learn version: ", sklearn.__version__)
print("matplotlib version: ", matplotlib.__version__)<jupyter_output>tensorflow version:  2.3.0
numpy version:  1.19.1
scikit learn version:  0.23.2
matplotlib version:  3.3.0<jupyter_text>1. Load data & preprocessingIn this notebook, I use pre-defined mnist dataset.<jupyter_code>(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()
# Convert images into flattern array
X_train = X_train.reshape(X_train.shape[0], X_train.shape[1] * X_train.shape[2])
X_test = X_test.reshape(X_test.shape[0], X_test.shape[1] * X_test.shape[2])
# Normalize
X_train = X_train / 255.0
X_test = X_test / 255.0
# Convert into float32
X_train = np.float32(X_train)
X_test = np.float32(X_test)
# Split original training dataset into train/validation dataset
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, shuffle=True)<jupyter_output><empty_output><jupyter_text>2. Create tf.data.Dataset<jupyter_code>train_batch_size = 64
test_batch_size = 1
n_classes = 10

# Build source dataset for training
X_train_dataset = tf.data.Dataset.from_tensor_slices(X_train)
y_train_dataset = tf.data.Dataset.from_tensor_slices(y_train).map(lambda x : tf.cast(tf.one_hot(x, n_classes), tf.float32))
train_dataset = tf.data.Dataset.zip((X_train_dataset, y_train_dataset)).batch(train_batch_size).repeat()

# Build source dataset for validation
X_valid_dataset = tf.data.Dataset.from_tensor_slices(X_val)
y_valid_dataset = tf.data.Dataset.from_tensor_slices(y_val).map(lambda x : tf.cast(tf.one_hot(x, n_classes), tf.float32))
validation_dataset = tf.data.Dataset.zip((X_valid_dataset, y_valid_dataset)).batch(train_batch_size).repeat()

# Build source dataset for test
X_test_dataset = tf.data.Dataset.from_tensor_slices(X_test)
y_test_dataset = tf.data.Dataset.from_tensor_slices(y_test).map(lambda x : tf.cast(tf.one_hot(x, n_classes), tf.float32))
test_dataset = tf.data.Dataset.zip((X_test_dataset, y_test_dataset)).batch(test_batch_size).repeat()<jupyter_output><empty_output><jupyter_text>Debug dataset<jupyter_code>def visualize_batch(X_batch, y_batch, y_pred=None):
    """visualize 1D ndarray"""
    assert len(X_batch) == len(y_batch)
    
    n_col = 10
    if len(X_batch) % n_col ==0:
        n_row=len(X_batch)//n_col
    else:
        n_row=len(X_batch)//n_col + 1
    
    fig = plt.figure(figsize=(20,15))
    
    for idx in range(len(y_batch)):
        if y_pred is not None:
            ax = fig.add_subplot(n_row, n_col, idx+1, title="gt={}, pred={}".format(np.argmax(y_batch[idx]), y_pred[idx]))
        else:
            ax = fig.add_subplot(n_row, n_col, idx+1, title="gt={}".format(np.argmax(y_batch[idx])))
        ax.imshow(X_batch[idx].reshape(28,28), cmap='gray')
        ax.axes.xaxis.set_visible(False)
        ax.axes.yaxis.set_visible(False)
    plt.show()
for X_batch, y_batch in train_dataset:
    visualize_batch(X_batch.numpy(), y_batch.numpy())
    break<jupyter_output><empty_output><jupyter_text>3. Create MLP modelNetwork structure (2 hidden layers):  FC(256) - FC(512) - FC(10)<jupyter_code>n_dense1 = 256
n_dense2 = 512
n_dense3 = n_classes
class MLP(tf.keras.Model):
    def __init__(self, units1, units2, units3):
        super(MLP, self).__init__(name="MLP")
        self.dense1 = tf.keras.layers.Dense(units1, activation=tf.nn.relu, kernel_initializer=tf.initializers.truncated_normal(stddev=0.1))
        self.dense2 = tf.keras.layers.Dense(units2, activation=tf.nn.relu, kernel_initializer=tf.initializers.truncated_normal(stddev=0.1))
        self.dense3 = tf.keras.layers.Dense(units3, activation=None, kernel_initializer=tf.initializers.truncated_normal(stddev=0.1))
    
    def call(self, inputs):
        x = self.dense1(inputs)
        x = self.dense2(x)
        x = self.dense3(x)
        return x<jupyter_output><empty_output><jupyter_text>4. Training<jupyter_code>lr = 0.0001
iter_per_epoch = X_train.shape[0] // train_batch_size
epochs = 15
checkpoint_path = './checkpoints_1/model_{epoch:04d}.ckpt'

model = MLP(n_dense1, n_dense2, n_dense3)
model.compile(optimizer=tf.keras.optimizers.Adam(lr),
              loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), 
              metrics=['accuracy'])

callbacks = [
    tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3),
    tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path, save_freq='epoch'),
]

model.fit(train_dataset, epochs=15, steps_per_epoch=int(len(X_train)/train_batch_size),
          validation_data=validation_dataset, validation_steps=int(len(X_val)/train_batch_size), callbacks=callbacks)<jupyter_output>Epoch 1/15
750/750 [==============================] - 2s 3ms/step - loss: 0.1271 - accuracy: 0.8073 - val_loss: 0.0611 - val_accuracy: 0.9165
Epoch 2/15
750/750 [==============================] - 2s 3ms/step - loss: 0.0493 - accuracy: 0.9312 - val_loss: 0.0441 - val_accuracy: 0.9403
Epoch 3/15
750/750 [==============================] - 2s 3ms/step - loss: 0.0370 - accuracy: 0.9493 - val_loss: 0.0362 - val_accuracy: 0.9493
Epoch 4/15
750/750 [==============================] - 2s 3ms/step - loss: 0.0301 - accuracy: 0.9600 - val_loss: 0.0314 - val_accuracy: 0.9554
Epoch 5/15
750/750 [==============================] - 2s 3ms/step - loss: 0.0254 - accuracy: 0.9668 - val_loss: 0.0282 - val_accuracy: 0.9601
Epoch 6/15
750/750 [==============================] - 2s 3ms/step - loss: 0.0219 - accuracy: 0.9717 - val_loss: 0.0259 - val_accuracy: 0.9627
Epoch 7/15
750/750 [==============================] - 2s 3ms/step - loss: 0.0191 - accuracy: 0.9761 - val_loss: 0.0242 - val_accuracy: 0.9651
Epoch [...]<jupyter_text>5. Test<jupyter_code>model.evaluate(test_dataset, steps=len(y_test))
index = np.random.choice(np.arange(0, len(y_test)), size=60)
test_input = X_test[index]
y_true = tf.keras.utils.to_categorical(y_test[index], num_classes=n_classes)

predicted = model.predict(test_input)
predicted_label = np.argmax(predicted, axis=1)

visualize_batch(test_input, y_true, predicted_label)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1587516960651289, 'fraction_numerical': 0.05140961857379768, 'mean_word_length': 5.014505893019039, 'pattern_counts': {'":': 0, '<': 27, '<?xml version=': 0, '>': 27, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '48907', 'license': 'MIT', 'n_tokens_mistral': 2693, 'n_tokens_neox': 2473, 'n_words': 439, 'path': 'notebooks_tf2/2_NeuralNetwork/1_BuiltInTrainingLoops.ipynb', 'repo_name': 'kattoyoshi/Tensorflow-Examples', 'size': '6633'}