<jupyter_start><jupyter_text>Additional callbacksIn this reading we'll be looking at more of the inbuilt callbacks available in Keras.<jupyter_code>import tensorflow as tf
print(tf.__version__)<jupyter_output>2.0.0<jupyter_text>We will again be using the sklearn diabetes dataset to demonstrate these callbacks.<jupyter_code># Load the diabetes dataset

from sklearn.datasets import load_diabetes

diabetes_dataset = load_diabetes()
# Save the input and target variables

from sklearn.model_selection import train_test_split

data = diabetes_dataset['data']
targets = diabetes_dataset['target']
# Split the data set into training and test sets

train_data, test_data, train_targets, test_targets = train_test_split(data, targets, test_size=0.1)<jupyter_output><empty_output><jupyter_text>Let's also build a simple model to fit to the data with our callbacks.<jupyter_code># Build the model

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense

model = tf.keras.Sequential([
    Dense(128, activation='relu', input_shape=(train_data.shape[1],)),
    Dense(64,activation='relu'),
    Dense(64, activation='relu'),
    Dense(64, activation='relu'),
    Dense(1)        
])
# Compile the model

model.compile(loss='mse',
                optimizer="adam",metrics=["mse","mae"])<jupyter_output><empty_output><jupyter_text>Now onto the callbacks!  Learning rate scheduler**Usage:** `tf.keras.callbacks.LearningRateScheduler(schedule, verbose=0)`The learning rate scheduler that we implemented in the previous reading as a custom callback is also available as a built in callback. As in our custom callback, the `LearningRateScheduler` in Keras takes a function `schedule` as an argument. This function `schedule` should take two arguments:* The current epoch (as an integer), and* The current learning rate,and return new learning rate for that epoch. The `LearningRateScheduler` also has an optional `verbose` argument, which prints information about the learning rate if it is set to 1.Let's see a simple example.<jupyter_code># Define the learning rate schedule function

def lr_function(epoch, lr):
    if epoch % 2 == 0:
        return lr
    else:
        return lr + epoch/1000
# Train the model

history = model.fit(train_data, train_targets, epochs=10,
                    callbacks=[tf.keras.callbacks.LearningRateScheduler(lr_function, verbose=1)], verbose=False)<jupyter_output>Epoch 00001: LearningRateScheduler reducing learning rate to 0.0010000000474974513.

Epoch 00002: LearningRateScheduler reducing learning rate to 0.0020000000474974513.

Epoch 00003: LearningRateScheduler reducing learning rate to 0.0020000000949949026.

Epoch 00004: LearningRateScheduler reducing learning rate to 0.005000000094994903.

Epoch 00005: LearningRateScheduler reducing learning rate to 0.004999999888241291.

Epoch 00006: LearningRateScheduler reducing learning rate to 0.009999999888241292.

Epoch 00007: LearningRateScheduler reducing learning rate to 0.009999999776482582.

Epoch 00008: LearningRateScheduler reducing learning rate to 0.01699999977648258.

Epoch 00009: LearningRateScheduler reducing learning rate to 0.016999999061226845.

Epoch 00010: LearningRateScheduler reducing learning rate to 0.025999999061226846.<jupyter_text>You can also use lambda functions to define your `schedule` given an epoch.<jupyter_code># Train the model with a difference schedule

history = model.fit(train_data, train_targets, epochs=10,
                    callbacks=[tf.keras.callbacks.LearningRateScheduler(lambda x:1/(3+5*x), verbose=1)], 
                    verbose=False)<jupyter_output>Epoch 00001: LearningRateScheduler reducing learning rate to 0.3333333333333333.

Epoch 00002: LearningRateScheduler reducing learning rate to 0.125.

Epoch 00003: LearningRateScheduler reducing learning rate to 0.07692307692307693.

Epoch 00004: LearningRateScheduler reducing learning rate to 0.05555555555555555.

Epoch 00005: LearningRateScheduler reducing learning rate to 0.043478260869565216.

Epoch 00006: LearningRateScheduler reducing learning rate to 0.03571428571428571.

Epoch 00007: LearningRateScheduler reducing learning rate to 0.030303030303030304.

Epoch 00008: LearningRateScheduler reducing learning rate to 0.02631578947368421.

Epoch 00009: LearningRateScheduler reducing learning rate to 0.023255813953488372.

Epoch 00010: LearningRateScheduler reducing learning rate to 0.020833333333333332.<jupyter_text>CSV logger**Usage** `tf.keras.callbacks.CSVLogger(filename, separator=',', append=False)`This callback streams the results from each epoch into a CSV file.The first line of the CSV file will be the names of pieces of information recorded on each subsequent line, beginning with the epoch and loss value. The values of metrics at the end of each epoch will also be recorded.The only compulsory argument is the `filename` for the log to be streamed to. This could also be a filepath.You can also specify the `separator` to be used between entries on each line.The `append` argument allows you the option to append your results to an existing file with the same name. This can be particularly useful if you are continuing training.Let's see an example.<jupyter_code># Train the model with a CSV logger

history = model.fit(train_data, train_targets, epochs=10,
                    callbacks=[tf.keras.callbacks.CSVLogger("results.csv")], verbose=False)<jupyter_output><empty_output><jupyter_text>Let's view the information in the CSV file we have created using `pandas`.<jupyter_code># Load the CSV

import pandas as pd

pd.read_csv("results.csv", index_col='epoch')<jupyter_output><empty_output><jupyter_text>Lambda callbacks**Usage** `tf.keras.callbacks.LambdaCallback(        on_epoch_begin=None, on_epoch_end=None,         on_batch_begin=None, on_batch_end=None,         on_train_begin=None, on_train_end=None)`Lambda callbacks are used to quickly define simple custom callbacks with the use of lambda functions.Each of the functions require some positional arguments.* `on_epoch_begin` and `on_epoch_end` expect two arguments: `epoch` and `logs`,* `on_batch_begin` and `on_batch_end` expect two arguments: `batch` and `logs` and* `on_train_begin` and `on_train_end` expect one argument: `logs`.Let's see an example of this in practice.<jupyter_code># Print the epoch number at the beginning of each epoch

epoch_callback = tf.keras.callbacks.LambdaCallback(
    on_epoch_begin=lambda epoch,logs: print('Starting Epoch {}!'.format(epoch+1)))
# Print the loss at the end of each batch

batch_loss_callback = tf.keras.callbacks.LambdaCallback(
    on_batch_end=lambda batch,logs: print('\n After batch {}, the loss is {:7.2f}.'.format(batch, logs['loss'])))
# Inform that training is finished

train_finish_callback = tf.keras.callbacks.LambdaCallback(
    on_train_end=lambda logs: print('Training finished!'))
# Train the model with the lambda callbacks

history = model.fit(train_data, train_targets, epochs=5, batch_size=100,
                    callbacks=[epoch_callback, batch_loss_callback,train_finish_callback], verbose=False)<jupyter_output>Starting Epoch 1!

 After batch 0, the loss is 2428.44.

 After batch 1, the loss is 2596.73.

 After batch 2, the loss is 2761.90.

 After batch 3, the loss is 3063.07.
Starting Epoch 2!

 After batch 0, the loss is 2772.30.

 After batch 1, the loss is 2323.33.

 After batch 2, the loss is 2988.37.

 After batch 3, the loss is 2807.04.
Starting Epoch 3!

 After batch 0, the loss is 3437.59.

 After batch 1, the loss is 2456.55.

 After batch 2, the loss is 2419.80.

 After batch 3, the loss is 2518.57.
Starting Epoch 4!

 After batch 0, the loss is 2802.71.

 After batch 1, the loss is 2655.53.

 After batch 2, the loss is 2411.85.

 After batch 3, the loss is 2981.86.
Starting Epoch 5!

 After batch 0, the loss is 2643.84.

 After batch 1, the loss is 2806.53.

 After batch 2, the loss is 2546.89.

 After batch 3, the loss is 2912.53.
Training finished!<jupyter_text>Reduce learning rate on plateau**Usage** `tf.keras.callbacks.ReduceLROnPlateau(            monitor='val_loss',             factor=0.1,             patience=10,             verbose=0,             mode='auto',             min_delta=0.0001,             cooldown=0,             min_lr=0)`The `ReduceLROnPlateau` callback allows reduction of the learning rate when a metric has stopped improving. The arguments are similar to those used in the `EarlyStopping` callback.* The argument `monitor` is used to specify which metric to base the callback on.* The `factor` is the factor by which the learning rate decreases i.e., new_lr=factor*old_lr.* The `patience` is the number of epochs where there is no improvement on the monitored metric before the learning rate is reduced.* The `verbose` argument will produce progress messages when set to 1.* The `mode` determines whether the learning rate will decrease when the monitored quantity stops increasing (`max`) or decreasing (`min`). The `auto` setting causes the callback to infer the mode from the monitored quantity.* The `min_delta` is the smallest change in the monitored quantity to be deemed an improvement.* The `cooldown` is the number of epochs to wait after the learning rate is changed before the callback resumes normal operation.* The `min_lr` is a lower bound on the learning rate that the callback will produce.Let's examine a final example.<jupyter_code># Train the model with the ReduceLROnPlateau callback

history = model.fit(train_data, train_targets, epochs=100, batch_size=100,
                    callbacks=[tf.keras.callbacks.ReduceLROnPlateau(
                        monitor="loss",factor=0.2, verbose=1)], verbose=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0753939643629622, 'fraction_numerical': 0.06900813678030693, 'mean_word_length': 4.755779490219324, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 9, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '47451', 'license': 'MIT', 'n_tokens_mistral': 3332, 'n_tokens_neox': 2777, 'n_words': 1059, 'path': 'Getting Started with TensorFlow 2/Week 3/Additional callbacks.ipynb', 'repo_name': 'pp-omega/TensorFlow2-for-Deep-Learning', 'size': '9709'}