<jupyter_start><jupyter_text>Tip. Click here to view this notebook on nbviewer.jupyter.org.     These notebooks are better read there, as Github default viewer ignores some of the formatting and interactive content.                             Hands-on Computer Vision with TensorFlow 2by <NAME> & <NAME> (Packt Pub.)        > Chapter 3: Modern Neural Networks    Notebook 3:Experimenting with Advanced Optimizers    Several modern algorithms derived from the stochastic gradient descent (SGD) are presented in Chapter 3 improving the optimization process. This notebook will showcase their usage and effects on the training of our CNN for digit classification.    Tip. The notebooks shared on this git repository illustrate some notions from the book "Hands-on Computer Vision with TensorFlow 2" written by <NAME> and <NAME>, published by Packt. If you enjoyed the insights shared here, please consider acquiring the book!The book provides further guidance for those eager to learn about computer vision and to harness the power of TensorFlow 2 and Keras to build efficient recognition systems for object detection, segmentation, video processing, smartphone applications, and more.                                    Leverage deep learning to create powerful image processing apps with TensorFlow 2 and Keras. Get the book for more insights!<jupyter_code>%matplotlib inline
import tensorflow as tf
import numpy as np
import matplotlib
from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>Preparing the Data We prepare the [MNIST](http://yann.lecun.com/exdb/mnist) dataset[$^1$](ref) (Yann LeCun and Corinna Cortes hold all copyrights for this dataset), as done in the previous notebooks:<jupyter_code>num_classes = 10
img_rows, img_cols, img_ch = 28, 28, 1
input_shape = (img_rows, img_cols, img_ch)

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

x_train = x_train.reshape(x_train.shape[0], *input_shape)
x_test = x_test.reshape(x_test.shape[0], *input_shape)

print('Training data: {}'.format(x_train.shape))
print('Testing data: {}'.format(x_test.shape))<jupyter_output>Training data: (60000, 28, 28, 1)
Testing data: (10000, 28, 28, 1)<jupyter_text>Preparing the Model We also reuse the _LeNet-5_ model[$^2$](ref) we previously implemented. However, to make the code more concise, we build this simple model using `tf.keras.Sequential`, a class inheriting from `tf.keras.Model` to create sequential architectures. Also we wrap the network's instantiation into a function, so that we can spawn multiple ones:<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten, Conv2D, MaxPooling2D
def lenet(name='lenet'):
    model = Sequential(name=name)
    # 1st block:
    model.add(Conv2D(6, kernel_size=(5, 5), padding='same', activation='relu', input_shape=input_shape))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    # 2nd block:
    model.add(Conv2D(16, kernel_size=(5, 5), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    # Dense layers:
    model.add(Flatten())
    model.add(Dense(120, activation='relu'))
    model.add(Dense(84, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))
    return model<jupyter_output><empty_output><jupyter_text>Training with Various Optimizers<jupyter_code>from tensorflow.keras import optimizers

# Setting some variables to format the logs:
log_begin_red, log_begin_blue, log_begin_green = '\033[91m', '\n\033[94m', '\033[92m'
log_begin_bold, log_begin_underline = '\033[1m', '\033[4m'
log_end_format = '\033[0m'<jupyter_output><empty_output><jupyter_text>To showcase the effect that modern optimizers (available in TensorFlow and Keras) have on trainings, we will create several similar LeNet instance, and train each with a different optimizer. First, we instantiate the various optimizers we want to compare[$^{3,4,5,6,7,8}$](ref):<jupyter_code>optimizers_examples = {
    'sgd': optimizers.SGD(),
    'momentum': optimizers.SGD(momentum=0.9),
    'nag': optimizers.SGD(momentum=0.9, nesterov=True),
    'adagrad': optimizers.Adagrad(),
    'adadelta': optimizers.Adadelta(),
    'rmsprop': optimizers.RMSprop(),
    'adam': optimizers.Adam()
}<jupyter_output><empty_output><jupyter_text>For each optimizer we are considering, we will instantiate a new LeNet model and train it with. We will save their training `history` (containing the losses and metrics history over training), for comparison *(this process takes time, especially on CPU!)*:<jupyter_code>history_per_optimizer = dict()

print("Experiment: {0}start{1} (training logs = off)".format(log_begin_red, log_end_format))
for optimizer_name in optimizers_examples:
    # Resetting the seeds (for random number generation), to reduce the impact of randomness on the comparison:
    tf.random.set_seed(42)
    np.random.seed(42)
    # Creating the model:
    model = lenet("lenet_{}".format(optimizer_name))
    optimizer = optimizers_examples[optimizer_name]
    model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])
    # Launching the training (we set `verbose=0`, so the training won't generate any logs):
    print("\t> Training with {0}: {1}start{2}".format(
        optimizer_name, log_begin_red, log_end_format))
    history = model.fit(x_train, y_train,
                        batch_size=32, epochs=20, validation_data=(x_test, y_test),
                        verbose=0)
    history_per_optimizer[optimizer_name] = history
    print('\t> Training with {0}: {1}done{2}.'.format(
        optimizer_name, log_begin_green, log_end_format))
print("Experiment: {0}done{1}".format(log_begin_green, log_end_format))<jupyter_output>Experiment: [91mstart[0m (training logs = off)
	> Training with sgd: [91mstart[0m
	> Training with sgd: [92mdone[0m.
	> Training with momentum: [91mstart[0m
	> Training with momentum: [92mdone[0m.
	> Training with nag: [91mstart[0m
	> Training with nag: [92mdone[0m.
	> Training with adagrad: [91mstart[0m
	> Training with adagrad: [92mdone[0m.
	> Training with adadelta: [91mstart[0m
	> Training with adadelta: [92mdone[0m.
	> Training with rmsprop: [91mstart[0m
	> Training with rmsprop: [92mdone[0m.
	> Training with adam: [91mstart[0m
	> Training with adam: [92mdone[0m.
Experiment: [92mdone[0m<jupyter_text>We finally plot the training and validation losses + accuracies, for comparison:<jupyter_code>fig, ax = plt.subplots(2, 2, figsize=(10,10), sharex='col')
ax[0, 0].set_title("loss")
ax[0, 1].set_title("val-loss")
ax[1, 0].set_title("accuracy")
ax[1, 1].set_title("val-accuracy")

lines, labels = [], []
for optimizer_name in history_per_optimizer:
    history = history_per_optimizer[optimizer_name]
    ax[0, 0].plot(history.history['loss'])
    ax[0, 1].plot(history.history['val_loss'])
    ax[1, 0].plot(history.history['accuracy'])
    line = ax[1, 1].plot(history.history['val_accuracy'])
    lines.append(line[0])
    labels.append(optimizer_name)

fig.legend(lines,labels, loc='center right', borderaxespad=0.1)
plt.subplots_adjust(right=0.85)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11619523017193566, 'fraction_numerical': 0.028286189683860232, 'mean_word_length': 5.223468507333909, 'pattern_counts': {'":': 0, '<': 31, '<?xml version=': 0, '>': 48, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '190713', 'license': 'MIT', 'n_tokens_mistral': 2502, 'n_tokens_neox': 2326, 'n_words': 667, 'path': 'Chapter03/ch3_nb3_experiment_with_optimizers.ipynb', 'repo_name': 'yellowjs0304/tf_computer_vision', 'size': '7244'}