<jupyter_start><jupyter_text>Tensorboard and Transfer Learning<jupyter_code># Load Some Data
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
y_test = np_utils.to_categorical(y_test)
y_train = np_utils.to_categorical(y_train)<jupyter_output><empty_output><jupyter_text>TensorboardTensorboard is a tool from Keras that can monitor the results of a tensorflow model and display it in a nice Tableau-like dashboard view. We can enable tensorboard and add it to our modelling process to get a better view of progress and save on some of the custom charting functions.  Create Model<jupyter_code># Set # of epochs
epochs = 10
def create_model():
  return tf.keras.models.Sequential([
    tf.keras.layers.Flatten(input_shape=(28, 28)),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dropout(0.2),
    tf.keras.layers.Dense(10, activation='softmax')
  ])
acc = keras.metrics.CategoricalAccuracy(name="accuracy")
pre = keras.metrics.Precision(name="precision")
rec = keras.metrics.Recall(name="recall")
metric_list = [acc, pre, rec]<jupyter_output><empty_output><jupyter_text>Add Tensorboard CallbackThe tensorboard can be added to the model as it is being fit as a callback. The primary parameter that matters there is the log_dir, where we can setup the folder to put the logs that the visualizations are made from. The example I have here is from the tensorflow documentation, generating a new subfolder for each execution. Using this to log the tensorboard data is fine, there's no need to change it without reason.   Launch TensorboardIn recent versions of VS Code, whioch I assume all of you have, tensorboard can be used directly in a VS Code tab:![VS Code Tensor](images/vscode_tensorboard.png "VS Code Tensor" )The command below launches tensorboard elsewhere, such as Google colab.Either way, the actual tensorboard feature works the same once launched.<jupyter_code>%load_ext tensorboard
%tensorboard --logdir logs/fit
# The logdir is wherever the logs are, this is specified in the callback setup. 
model = create_model()
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=metric_list)

log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.fit(x=x_train, 
          y=y_train, 
          epochs=epochs, 
          validation_data=(x_test, y_test), 
          callbacks=[tensorboard_callback])<jupyter_output>Epoch 1/2
1875/1875 [==============================] - 8s 4ms/step - loss: 0.2227 - accuracy: 0.9349 - precision: 0.9548 - recall: 0.9161 - val_loss: 0.1060 - val_accuracy: 0.9686 - val_precision: 0.9736 - val_recall: 0.9636
Epoch 2/2
1875/1875 [==============================] - 7s 4ms/step - loss: 0.0952 - accuracy: 0.9709 - precision: 0.9757 - recall: 0.9668 - val_loss: 0.0931 - val_accuracy: 0.9690 - val_precision: 0.9732 - val_recall: 0.9662<jupyter_text>Tensorboard ContentsThe first page of the tensorboard page gives us a nice pretty view of our training progress - this part should be quite straightforward.  Tensorboard ImagesWe can also use the tensorboard to visualize other stuff.<jupyter_code># Sets up a timestamped log directory.

logdir = "logs/train_data/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
# Creates a file writer for the log directory.
file_writer = tf.summary.create_file_writer(logdir)
with file_writer.as_default():
    # Don't forget to reshape.
    images = np.reshape(x_train[0:25], (-1, 28, 28, 1))
    tf.summary.image("25 training data examples", images, max_outputs=25, step=0)<jupyter_output><empty_output><jupyter_text>Using Pretrained ModelsAs we've seen lately, training neural networks can take a really long time. Highly accurate models such as the ones that are used for image recognition in a self driving cars can take multiple computers days or weeks to train. With one laptop we don't really have the ability to get anywhere close to that. Is there any hope of getting anywhere near that accurate?We can use models that have been trained on large datasets and adapt them to our purposes. By doing this we can benefit from all of that other learning that is embedded into a model without going through a training process that would be impossible with our limited resources. We will look at using a pretrained model here, and at making modifications to it next time.  Functional ModelsI have lied to you, I forgot that the pretrained models are not sequntial ones (generally, not as a rule), so some of the syntax here is for functional models. It leads to us using some slightly unfamiliar syntax.<jupyter_code>_URL = 'https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip'
path_to_zip = tf.keras.utils.get_file('cats_and_dogs.zip', origin=_URL, extract=True)
PATH = os.path.join(os.path.dirname(path_to_zip), 'cats_and_dogs_filtered')

train_dir = os.path.join(PATH, 'train')
validation_dir = os.path.join(PATH, 'validation')

BATCH_SIZE = 32
IMG_SIZE = (160, 160)

train_dataset = tf.keras.utils.image_dataset_from_directory(train_dir,
                                                            shuffle=True,
                                                            batch_size=BATCH_SIZE,
                                                            image_size=IMG_SIZE)
validation_dataset = tf.keras.utils.image_dataset_from_directory(validation_dir,
                                                                 shuffle=True,
                                                                 batch_size=BATCH_SIZE,
                                                                 image_size=IMG_SIZE)<jupyter_output>Found 2000 files belonging to 2 classes.<jupyter_text>Download ModelThere are several models that are pretrained and available to us to use. VGG16 is one developed to do image recognition, the name stands for "Visual Geometry Group" - a group of researchers at the University of Oxford who developed it, and ‘16’ implies that this architecture has 16 layers. The model got ~93% on the ImageNet test that we mentioned a couple of weeks ago. ![VGG16](images/vgg16.png "VGG16" )<jupyter_code>from keras.applications.vgg16 import VGG16
from keras.layers import Input
from keras.models import Model

input_tensor = Input(shape=(160, 160, 3))
vgg = VGG16(include_top=False, weights='imagenet', input_tensor=input_tensor)

for layer in vgg.layers:
    layer.trainable = False

x = Flatten()(vgg.output)
prediction = Dense(1, activation='sigmoid')(x)

model = Model(inputs=vgg.input, outputs=prediction)

model.summary()
model.compile(optimizer='adam',
              loss='categorical_crossentropy',
              metrics=metric_list)

log_dir = "logs/fit/VGG" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

model.fit(train_dataset, 
          epochs=epochs, 
          validation_data=validation_dataset, 
          callbacks=[tensorboard_callback])
          
model.evaluate(validation_dataset)
model.evaluate(validation_dataset)<jupyter_output>32/32 [==============================] - 176s 5s/step - loss: 0.0000e+00 - accuracy: 1.0000 - precision: 0.0000e+00 - recall: 0.0000e+00<jupyter_text>More Complex DataWe can use the rose data for a more complex dataset and a more interesting example in terms of accuracy.<jupyter_code>import pathlib
import PIL 

dataset_url = "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz"
data_dir = tf.keras.utils.get_file(origin=dataset_url,
                                   fname='flower_photos',
                                   untar=True)
data_dir = pathlib.Path(data_dir)

#Flowers
batch_size = 32
img_height = 180
img_width = 180

train_ds = tf.keras.utils.image_dataset_from_directory(
  data_dir,
  validation_split=0.2,
  subset="training",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)

val_ds = tf.keras.utils.image_dataset_from_directory(
  data_dir,
  validation_split=0.2,
  subset="validation",
  seed=123,
  image_size=(img_height, img_width),
  batch_size=batch_size)

class_names = train_ds.class_names
print(class_names)
input_tensor = Input(shape=(180, 180, 3))
vgg = VGG16(include_top=False, weights='imagenet', input_tensor=input_tensor)

for layer in vgg.layers:
    layer.trainable = False

x = Flatten()(vgg.output)
prediction = Dense(5)(x)

model = Model(inputs=vgg.input, outputs=prediction)

model.summary()
model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), 
            optimizer="adam", 
            metrics=keras.metrics.SparseCategoricalAccuracy(name="accuracy"))

log_dir = "logs/fit/VGG" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
callback = EarlyStopping(monitor='loss', patience=3, restore_best_weights=True) 

model.fit(train_ds,
            epochs=epochs,
            verbose=1,
            validation_data=val_ds,
            callbacks=[tensorboard_callback, callback])<jupyter_output>Epoch 1/2
92/92 [==============================] - 341s 4s/step - loss: 6.6734 - accuracy: 0.6999 - val_loss: 5.3380 - val_accuracy: 0.7820
Epoch 2/2
92/92 [==============================] - 330s 4s/step - loss: 1.1660 - accuracy: 0.9159 - val_loss: 4.9176 - val_accuracy: 0.8120
{'fraction_non_alphanumeric': 0.10776495278069255, 'fraction_numerical': 0.032214060860440714, 'mean_word_length': 3.8751918158567773, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 16, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '512343', 'license': 'MIT', 'n_tokens_mistral': 3147, 'n_tokens_neox': 2877, 'n_words': 833, 'path': 'img_opt_tensorboard.ipynb', 'repo_name': 'allyryry/3950_Workbooks', 'size': '9530'}