<jupyter_start><jupyter_text>OptimizersOn this notebook we will take a look at some of the different optimizers present in keras backend and will compare them in a Tensorboard visualization. The dataWe will use our old friend MNIST for its simplicity. Load the dataset and preprocess it. Keep in mind that we are going to use convolutions and it needs to be transformed into 4 dim tensors.Hint: use the expand_dims function from numpy<jupyter_code>import os, time
import os, time
from numpy import expand_dims
import tensorflow as tf
physical_devices = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

tf.keras.backend.clear_session() 

# the data, split between train and test sets
x_train = ...
x_test = ...

print('x_train shape:', x_train.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')<jupyter_output><empty_output><jupyter_text>Model ArchitectureWe are going to use convolutions on this example. Please don't be afraid, you will have a lot of convolution theory later on CNN block. Our model will consist on:- A conv2d layer with 16 units and 3x3 conv shape, relu-activated.- A maxPool layer- A conv2d layer with 16 units and 3x3 conv shape, relu-activated.- A maxPool layer- A flatten layer, followed by a dense softmax activated with the amount of categories as the number of units.- Our optimizer will be 'SGD' and we will optimize sparse categorical crossentropy. Add accuracy as a metric. Build the model<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense

optimizer = 'SGD'

...
model.summary()<jupyter_output><empty_output><jupyter_text>Train the model for 5 epochs, with a batch size of 32. Evaluate its performance<jupyter_code>...<jupyter_output><empty_output><jupyter_text>Optimizer benchmarkWe are going to compare more than one optimizer on the same problem.  Create a build_model function with the optimizer as a parameter and the same model architecture. Try the one you want<jupyter_code>def build_model(optimizer):
    ...
    return model<jupyter_output><empty_output><jupyter_text>Now let's run the model with all the optimizers in the list and view the results in tensorboard Let's do precisely that!     Hint: remember to add the tensorboard as a callback for the training.     Hint2: use the function os.path.join to include the optimizer name on each model call<jupyter_code>from tensorflow.keras.callbacks import TensorBoard
log_path = '/home/fer/data/formaciones/afi/tensorboard_log/optimizers_experiment'
optimizers = [
    'Adadelta',
    'Adagrad',
    'Adam',
    'Adamax',
    'Nadam',
    'RMSprop',
    'SGD']
for ...<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07235421166306695, 'fraction_numerical': 0.007199424046076314, 'mean_word_length': 5.041304347826087, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '37604', 'license': 'Apache-2.0', 'n_tokens_mistral': 842, 'n_tokens_neox': 805, 'n_words': 328, 'path': 'Training/OptimizersTraining.ipynb', 'repo_name': 'fermaat/afi_deep_learning_intro', 'size': '2778'}