<jupyter_start><jupyter_text>Deep Learning from Pre-Trained Models with Keras IntroductionImageNet, an image recognition benchmark dataset*, helped trigger the modern AI explosion.  In 2012, the AlexNet architecture (a deep convolutional-neural-network) rocked the ImageNet benchmark competition, handily beating the next best entrant.  By 2014, all the leading competitors were deep learning based.  Since then, accuracy scores continued to improve, eventually surpassing human performance.In this hands-on tutorial, and later exercise, we will build on this pioneering work to create our own neural-network architecture for image recognition.  Participants will use the elegant Keras deep learning programming interface to build and train TensorFlow models for image classification tasks on the CIFAR-10 / MNIST datasets*.  We will demonstrate the use of transfer learning* (to give our networks a head-start by building on top of existing, ImageNet pre-trained, network layers*), and explore how to improve model performance for standard deep learning pipelines.  We will use cloud-based interactive Jupyter notebooks to work through our explorations step-by-step.This tutorial is designed as an introduction to the topic for a general, but technical audience. As a practical introduction, it will focus on tools and their application. Previous ML (Machine Learning) experience is not required; but, previous experience with scripting in Python will help. Participants are expected to bring their own laptops and sign-up for free online cloud services (e.g., Google Colab, Kaggle).  They may also need to download free, open-source software prior to arriving for the workshop.This tutorial assumes some basic knowledge of neural networks. If you’re not already familiar with neural networks, then you can learn the basics concepts behind neural networks at [course.fast.ai](https://course.fast.ai/).* Tutorial materials are derived from:  * [PyTorch Tutorials](https://github.com/kaust-vislab/pytorch-tutorials) by David Pugh.  * [What is torch.nn really?](https://pytorch.org/tutorials/beginner/nn_tutorial.html) by <NAME>, <NAME>, <NAME>.  * [Machine Learning Notebooks](https://github.com/ageron/handson-ml2) (2nd Ed.) by <NAME>.  * *Deep Learning with Python* by <NAME>.  Jupyter NotebooksThis is a Jupyter Notebook.  It provides a simple, cell-based, IDE for developing and exploring complex ideas via code, visualizations, and documentation.A notebook has two primary types of cells: i) `markdown` cells for textual notes and documentation, such as the one you are reading now, and ii) `code` cells, which contain snippets of code (typically *Python*, but also *bash* scripts) that can be executed.  The currently selected cell appears within a box. A green box indicates that the cell is editable.  Clicking inside a *code* cell makes it selected and editable.  Double-click inside *markdown* cells to edit.Use `Tab` for context-sensitive code-completion assistance when editing Python code in *code* cells.  For example, use code assistance after a `.` seperator to find available object members.  For help documentation, create a new *code* cell, and use commands like `dir(`*module*`)`, `help(`*topic*`)`, `?`*name*, or `??`*function* for user provided *module*, *topic*, variable *name*, or *function* name.  The magic `?` and `??` commands show documentation / source code in a separate pane.Clicking on `[Run]` or pressing `Ctrl-Enter` will execute the contents of a cell.  A *markdown* cell converts to its display version, and a *code* cell runs the code inside.  To the left of a *code* cell is a small text bracket `In [ ]:`.  If the bracket contains an asterix, e.g., `In [*]:`, that cell is currently executing.  Only one cell executes at a time (if multiple cells are *Run*, they are queued up to execute in the order they were run).  When a *code* cell finishes executing, the bracket shows an execution count in the bracket – each *code* cell execution increments the counter and provides a way to determine the order in which codes were executed – e.g., `In [7]` for the seventh cell to complete.  The output produced by a *code* cell appears at the bottom of that cell after it executes.  The output generated by a code cell includes anything printed to the output during execution (e.g., print statements, or thrown errors) and the final value generated by the cell (i.e., not the intermediate values).  The final value is 'pretty printed' by Jupyter.Typically, notebooks are written to be executed in order, from top to bottom.  Behind the scenes, however, each Notebook has a single Python state (the `kernel`), and each *code* cell that executes, modifies that state.  It is possible to modify and re-run earlier cells; however, care must be taken to also re-run any other cells that depend upon the modified one.  List the Python state global variables with the magic command `%wgets`.  The *kernel* can be restarted to a known state, and cell output cleared, if the Python state becomes too confusing to fix manually (choose `Restart & Clear Output` from the Jupyter `Kernel` menu) – this requires running each *code* cell again.Complete user documentation is available at [jupyter-notebook.readthedocs.io](https://jupyter-notebook.readthedocs.io/en/stable/notebook.htmlnotebook-user-interface). Many helpful tips and techniques from [28 Jupyter Notebook Tips, Tricks, and Shortcuts](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/).  Setup  Setup ColabIn order to run this notebook in [Google Colab](https://colab.research.google.com) you will need a [Google Account](https://accounts.google.com/).  Sign-in to your Google account, if necessary, and then start the notebook.Change Google Colab runtime to use GPU:* Click `Runtime` -> `Change runtime type` menu item* Specify `Hardware accelerator` as `GPU`* Click **[Save]** buttonThe session indicator (toolbar / status ribbon under menu) should briefly appear as `Connecting...`.  When the session restarts, continue with the next cell (specifying TensorFlow version v2.x):<jupyter_code>try:
  # %tensorflow_version only exists in Colab.
  %tensorflow_version 2.x
except Exception:
  pass<jupyter_output><empty_output><jupyter_text>Download DataThere are two image datasets ([CIFAR10](https://www.cs.toronto.edu/~kriz/cifar.html) and [MNIST](http://yann.lecun.com/exdb/mnist/index.html)) which these tutorial / exercise notebooks use.These datasets are available from a variety of sources, including this repository – depending on how the notebook was launched (e.g., Git+LFS / Binder contains entire repository, Google Colab only contains the notebook).Because data is the fundamental fuel for deep learning, we need to ensure the required datasets for this tutorial are available to the current notebook session.  The following steps will ensure the data is already available (or downloaded), and cached where Keras can find them.<jupyter_code># %load cache_utils.py
import pathlib
import tensorflow.keras.utils as Kutils

def cache_mnist_data():
  for n in ["mnist.npz", "kaggle/train.csv", "kaggle/test.csv"]:
    path = pathlib.Path("../datasets/mnist/%s" % n).absolute()
    if not path.is_file():
      print("Skipping: missing local dataset file: %s" % n)
    else:
      DATA_URL = "file:///" + str(path)
      try:
        data_file_path = Kutils.get_file(n.replace('/','-mnist-'), DATA_URL)
        print("Cached file: %s" % n)
      except (FileNotFoundError, ValueError, Exception) as e:
        print("Cache Failed: First fetch file: %s" % n)

def cache_cifar10_data():
  for n in ["cifar-10.npz", "cifar-10-batches-py.tar.gz"]:
      path = pathlib.Path("../datasets/cifar10/%s" % n).absolute()
      if not path.is_file():
        print("Skipping: missing local dataset file: %s" % n)
      else:
        DATA_URL = "file:///" + str(path)
        try:
          data_file_path = Kutils.get_file(n, DATA_URL)
          print("Cached file: %s" % n)
        except (FileNotFoundError, ValueError, Exception) as e:
          print("Cache Failed: First fetch file: %s" % n)

def cache_models():
  for n in ["vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5"]:
    path = pathlib.Path("../models/%s" % n).absolute()
    if not path.is_file():
      print("Skipping: missing local dataset file: %s" % n)
    else:
      DATA_URL = "file:///" + str(path)
      try: 
        data_file_path = Kutils.get_file(n, DATA_URL, cache_subdir='models')
        print("Cached file: %s" % n)
      except (FileNotFoundError, ValueError, Exception) as e:
        print("Cache Failed: First fetch file: %s" % n)<jupyter_output><empty_output><jupyter_text>Follow the instructions below and run just the appropriate cells needed to acquire the required datasets:  Download CIFAR10 DataIf you are using Binder to run this notebook, then the data is already downloaded and available.  Skip to the next step.If you are using Google Colab to run this notebook, then you will need to download the data before proceeding.  Download CIFAR10 with KerasIf you are running this notebook using Google Colab, then download the data using the Keras `load_data()` API by running the code in the following cell.<jupyter_code>from tensorflow.keras.datasets import cifar10
cache_cifar10_data()
cifar10.load_data();<jupyter_output><empty_output><jupyter_text>Tutorial SetupInitialize the Python environment by importing and verifying the modules we will use.<jupyter_code>import os
import sys
import pathlib
import random
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import tensorflow.keras as keras<jupyter_output><empty_output><jupyter_text>`%matplotlib inline` is a magic command that makes *matplotlib* charts and plots appear was outputs in the notebook.`%matplotlib notebook` enables semi-interactive plots that can be enlarged, zoomed, and cropped while the plot is active.  One issue with this option is that new plots appear in the active plot widget, not in the cell where the data was produced.<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Now check the runtime environment to ensure it can run this notebook.  If there is an `Exception`, or if there are no GPUs, you will need to run this notebook in a more capable environment (see `README.md`, or ask instructor for additional help).<jupyter_code># %load verify_runtime.py
# Verify runtime environment

try:
    # %tensorflow_version only exists in Colab.
    %tensorflow_version 2.x
    IS_COLAB = True
except Exception:
    IS_COLAB = False
print("is_colab:", IS_COLAB)

assert tf.__version__ >= "2.0", "TensorFlow version >= 2.0 required."
print("tensorflow_version:", tf.__version__)

assert sys.version_info >= (3, 5), "Python >= 3.5 required."
print("python_version:", "%s.%s.%s-%s" % (sys.version_info.major, 
                                          sys.version_info.minor,
                                          sys.version_info.micro,
                                          sys.version_info.releaselevel
                                         ))

print("executing_eagerly:", tf.executing_eagerly())

try:
    __physical_devices = tf.config.list_physical_devices('GPU')
except AttributeError:
    __physical_devices = tf.config.experimental.list_physical_devices('GPU')
    
if len(__physical_devices) == 0:
    print("No GPUs available. Expect training to be very slow.")
    if IS_COLAB:
        print("Go to `Runtime` > `Change runtime` and select a GPU hardware accelerator."
              "Then `Save` to restart session.")
else:
    print("is_built_with_cuda:", tf.test.is_built_with_cuda())
    print("gpus_available:", [d.name for d in __physical_devices])<jupyter_output><empty_output><jupyter_text>CIFAR10 - Dataset ProcessingThe previously acquired CIFAR10 dataset is the essential input needed to train an image classification model. Before using the dataset, there are several preprocessing steps required to load the data, and create the correctly sized training, validation, and testing arrays used as input to the network.The following data preparation steps are needed before they can become inputs to the network:* Cache the downloaded dataset (to assist Keras `load_data()` functionality).* Load the dataset (CIFAR10 is small, and fits into a `numpy` array).* Verify the shape and type of the data, and understand it...* Convert label indices into categorical vectors.* Convert image data from integer to float values, and normalize.  * Verify converted input data.  Cache DataMake downloaded data available to Keras (and check if it's really there).  Provide dataset utility functions.__Note__: We are ready to begin if the `find` command shows a found data file.<jupyter_code># Cache CIFAR10 Datasets
cache_cifar10_data()
%%bash
find ~/.keras -name "cifar-10*" -type f<jupyter_output><empty_output><jupyter_text>These helper function assist with managing the three label representations we will encounter:* label index: a number representing a class* label names: a *human readable* text representation of a class* category vector: a vector space to represent the categoriesThe label index `1` represents an `automobile`, and `2` represents a `bird`; but, `1.5` doesn't make a `bird-mobile`.  We need a representation where each dimension is a continuum of that feature.  There are 10 distinct categories, so we encode them as a 10-dimensional vector space, where the i-th dimension represents the i-th class.  An `automobile` becomes `[0,1,0,0,0,0,0,0,0,0]`, a `bird` becomes `[0,0,1,0,0,0,0,0,0,0]` (these are called *one-hot encodings*), and a `bird-mobile` (which we couldn't represent previously) can be encoded as `[0,0.5,0.5,0,0,0,0,0,0,0]`.**Note:** We already know how our dataset is represented.  Typically, one would load the data first, analyse the class representation, and then write the helper functions.<jupyter_code># Helper functionality to provide human-readable labels
cifar10_label_names = ['airplane', 'automobile', 
                       'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 
                       'ship', 'truck']

def cifar10_index_label(idx):
    return cifar10_label_names[int(idx)]

def cifar10_category_label(cat):
    return cifar10_index_label(cat.argmax())

def cifar10_label(v):
    return cifar10_index_label(v) if np.isscalar(v) or np.size(v) == 1 else cifar10_category_label(v)<jupyter_output><empty_output><jupyter_text>Load DataDatasets for classification require two parts: i) the input data (`x` in our nomenclature), and ii) the labels (`y`).  Classifiction takes an `x` as input, and returns a `y` (the class) as output.When training a model from a dataset (called the `train`ing dataset), it is important to keep some of the data aside (called the `test` set).  If we didn't, the model could just memorize the data without learning a generalization that would apply to novel related data.  The `test` set is used to evaluate the typical real performance of the model.<jupyter_code>from tensorflow.keras.datasets import cifar10

# The data, split between train and test sets:
(x_train, y_train), (x_test, y_test) = cifar10.load_data()<jupyter_output><empty_output><jupyter_text>**Note:** Backup plan: Run the following cell if the data didn't load via `cifar10.load_data` above.<jupyter_code># Try secondary data source if the first didn't work
try:
    print("data loaded." if type((x_train, y_train, x_test, y_test)) else "load failed...")
except NameError:
    with np.load('../datasets/cifar10/cifar-10.npz') as data:
        x_train = data['x_train']
        y_train = data['y_train']
        x_test = data['x_test']
        y_test = data['y_test']
    print("alternate data load." if type((x_train, y_train, x_test, y_test)) else "failed...")<jupyter_output><empty_output><jupyter_text>Explore DataExplore data types, shape, and value ranges.  Ensure they make sense, and you understand the data well.<jupyter_code>print('x_train type:', type(x_train), ',', 'y_train type:', type(y_train))
print('x_train dtype:', x_train.dtype, ',', 'y_train dtype:', y_train.dtype)
print('x_train shape:', x_train.shape, ',', 'y_train shape:', y_train.shape)
print('x_test shape:', x_test.shape, ',', 'y_test shape:', y_test.shape)
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
print('x_train (min, max, mean): (%s, %s, %s)' % (x_train.min(), x_train.max(), x_train.mean()))
print('y_train (min, max): (%s, %s)' % (y_train.min(), y_train.max()))<jupyter_output><empty_output><jupyter_text>* The data is stored in Numpy arrays.* The datatype for both input data and labels is a small unsigned int.  They represent different things though.  The input data represents pixel value, the labels represent the category.* There are 50000 training data samples, and 10000 testing samples.* Each input sample is a colour images of 32x32 pixels, with 3 channels of colour (RGB), for a total size of 3072 bytes.  Each label sample is a single byte.  * A 32x32 pixel, 3-channel colour image (2-D) can be represented as a point in a 3072 dimensional vector space.* We can see that pixel values range between 0-255 (that is the range of `uint8`) and the mean value is close to the middle.  The label values range between 0-9, which corresponds to the 10 categories the labels represent.Lets explore the dataset visually, looking at some actual images, and get a statistical overview of the data.Most of the code in the plotting function below is there to tweak the appearance of the output.  The key functionality comes from `matplotlib` functions `imshow` and `hist`, and `numpy` function `histogram`.<jupyter_code>def cifar10_imageset_plot(img_data=None):
    (x_imgs, y_imgs) = img_data if img_data else (x_train, y_train)
    fig = plt.figure(figsize=(16,8))

    for i in range(40):
        plt.subplot(4, 10, i + 1)
        plt.xticks([])
        plt.yticks([])
        idx = int(random.uniform(0, x_imgs.shape[0]))
        plt.title(cifar10_label(y_imgs[idx]))
        plt.imshow(x_imgs[idx], cmap=plt.get_cmap('gray'))
    plt.show()
# Show array of random labelled images with matplotlib (re-run cell to see new examples)
cifar10_imageset_plot((x_train, y_train))
# %load histogram_utils.py
# Histogram utils

def histogram_plot(img_data=None):
  (x_data, y_data) = img_data if img_data else (x_train, y_train)

  fig = plt.figure(figsize=(12,5))

  plt.subplot(1,2,1)
  plt.hist(y_data, bins = range(int(y_data.min()), int(y_data.max() + 2)))
  plt.xticks(range(int(y_data.min()), int(y_data.max() + 2)))
  plt.title("y histogram")
  plt.subplot(1,2,2)
  plt.hist(x_data.flat, bins = range(int(x_data.min()), int(x_data.max() + 2)))
  plt.title("x histogram")
  plt.tight_layout()
  plt.show()

  hist, bins = np.histogram(y_data, bins = range(int(y_data.min()), int(y_data.max() + 2)))
  print('y histogram counts:', hist)

def histogram_label_plot(train_img_data=None, test_img_data=None):
  (x_train_data, y_train_data) = train_img_data if train_img_data else (x_train, y_train)
  (x_test_data, y_test_data) = test_img_data if test_img_data else (x_test, y_test)

  x_data_min = int(min(x_train_data.min(), x_test_data.min()))
  x_data_max = int(min(x_train_data.max(), x_test_data.max()))
  y_data_min = int(min(y_train_data.min(), y_test_data.min()))
  y_data_max = int(min(y_train_data.max(), y_test_data.max()))
  num_rows = y_data_max - y_data_min + 1
  
  fig = plt.figure(figsize=(12,12))

  plot_num = 1
  for lbl in range(y_data_min, y_data_max):
    plt.subplot(num_rows, 2 , plot_num)
    plt.hist(x_train_data[y_train_data.squeeze() == lbl].flat, bins = range(x_data_min, x_data_max + 2))
    plt.title("x train histogram - label %s" % lbl)
    plt.subplot(num_rows, 2 , plot_num + 1)
    plt.hist(x_test_data[y_test_data.squeeze() == lbl].flat, bins = range(x_data_min, x_data_max + 2))
    plt.title("x test histogram - label %s" % lbl)
    plot_num += 2

  plt.tight_layout(pad=0)
  plt.show()

histogram_plot((x_train, y_train))
histogram_plot((x_test, y_test))<jupyter_output><empty_output><jupyter_text>The data looks reasonable: there are sufficient examples for each category (y_train) and a near-normal distribution of pixel values that appears similar in both the train and test datasets.If there had been a significant imbalance in the number of examples in each category, test accuracy would be adversely affected (infrequent categories tend to get ignored). Use a tool like [`imbalanced-learn`](https://imbalanced-learn.org/stable/) to resample and re-balance the dataset.Lets do one more sanity check to ensure that the data distributions are also similar per-category.<jupyter_code>histogram_label_plot((x_train, y_train), (x_test, y_test))<jupyter_output><empty_output><jupyter_text>Again, the data looks reasonable. The distributions differ slightly between categories, but are similar between  the train and test datasets for a given category label.If there had been a significant difference in distributions, consider resampling the train and test datasets, or consider some kind of normalization as part of the training and inference data pipelines.  The next aspect of the input data to grapple with is how the input vector space corresponds with the output category space.  Is the correspondence simple, e.g., distances in the input space relate to distances in the output space; or, more complex.   Visualizing training samples using PCA[Principal Components Analysis (PCA)](https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) can be used as a visualization tool to see if there are any obvious patterns in the training samples.PCA re-represents the input data by changing the basis vectors that represent them.  These new orthonormal basis vectors (eigen vectors) represent variance in the data (ordered from largest to smallest).  Projecting the data samples onto the first few (2 or 3) dimensions will let us see the data with the biggest differences accounted for.The following cell uses `scikit-learn` to calculate PCA eigen vectors for a random subset of the data (10%).<jupyter_code>import sklearn
import sklearn.decomposition

_prng = np.random.RandomState(42)

pca = sklearn.decomposition.PCA(n_components=40, random_state=_prng)

x_train_flat = x_train.reshape(*x_train.shape[:1], -1)
y_train_flat = y_train.reshape(y_train.shape[0])
print("x_train:", x_train.shape, "y_train", y_train.shape)
print("x_train_flat:", x_train_flat.shape, "y_train_flat", y_train_flat.shape)
pca_train_features = pca.fit_transform(x_train_flat, y_train_flat)
print("pca_train_features:", pca_train_features.shape)

# Sample 10% of the PCA results
_idxs = _prng.randint(y_train_flat.shape[0], size=y_train_flat.shape[0] // 10)
pca_features = pca_train_features[_idxs]
pca_category = y_train_flat[_idxs]
print("pca_features:", pca_features.shape, 
      "pca_category", pca_category.shape, 
      "min,max category:", pca_category.min(), pca_category.max())
def pca_components_plot(components_, shape_=(32, 32, 3)):
    fig = plt.figure(figsize=(16,8))

    for i in range(min(40, components_.shape[0])):
        plt.subplot(4, 10, i + 1)
        plt.xticks([])
        plt.yticks([])
        eigen_vect = (components_[i] - np.min(components_[i])) / np.ptp(pca.components_[i])
        plt.title('component: %s' % i)
        plt.imshow(eigen_vect.reshape(shape_), cmap=plt.get_cmap('gray'))
    plt.show()<jupyter_output><empty_output><jupyter_text>This plot shows the new eigen vector basis functions suggested by the PCA analysis.  Any image in our dataset can be created as a linear combination of these basis vectors.  At a guess, the most prevalent feature of the dataset is that there is something at the centre of the image that is distinct from the background (components 0 & 2) and there is often a difference between 'land' & 'sky' (component 1) – compare with the sample images shown previously.<jupyter_code>pca_components_plot(pca.components_)<jupyter_output><empty_output><jupyter_text>These are 2D and 3D scatter plot functions that colour the points by their labels (so we can see if any 'clumps' of points correspond to actual categories).<jupyter_code>def category_scatter_plot(features, category, title='CIFAR10'):
    num_category = 1 + category.max() - category.min()

    fig, ax = plt.subplots(1, 1, figsize=(12, 10))
    cm = plt.cm.get_cmap('tab10', num_category)
    sc = ax.scatter(features[:,0], features[:,1], c=category, alpha=0.4, cmap=cm)
    ax.set_xlabel("Component 1")
    ax.set_ylabel("Component 2")
    ax.set_title(title)
    plt.colorbar(sc)
    plt.show()
from mpl_toolkits.mplot3d import Axes3D

def category_scatter3d_plot(features, category, title='CIFAR10'):
    num_category = 1 + category.max() - category.min()
    mean_feat = np.mean(features, axis=0)
    std_feat = np.std(features, axis=0)
    min_range = mean_feat - std_feat
    max_range = mean_feat + std_feat
    
    fig = plt.figure(figsize=(12, 10))
    cm = plt.cm.get_cmap('tab10', num_category)
    ax = fig.add_subplot(111, projection='3d')
    sc = ax.scatter(features[:,0], features[:,1], features[:,2],
                    c=category, alpha=0.85, cmap=cm)
    ax.set_xlabel("Component 1")
    ax.set_ylabel("Component 2")
    ax.set_zlabel("Component 3")
    ax.set_title(title)
    ax.set_xlim(2.0 * min_range[0], 2.0 * max_range[0])
    ax.set_ylim(2.0 * min_range[1], 2.0 * max_range[1])
    ax.set_zlim(2.0 * min_range[2], 2.0 * max_range[2])
    plt.colorbar(sc)
    plt.show()
category_scatter_plot(pca_features, pca_category, title='CIFAR10 - PCA')<jupyter_output><empty_output><jupyter_text>**Note:** 3D PCA plot works best with `%matplotlib notebook` to enable interactive rotation (enabled at start of session).<jupyter_code>category_scatter3d_plot(pca_features, pca_category, title='CIFAR10 - PCA')<jupyter_output><empty_output><jupyter_text>The data in its original image space does not appear to cluster into corresponding categories. Visualizing training sample using t-SNE[t-distributed Stochastic Neighbor Embedding (t-SNE)](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.htmlsklearn.manifold.TSNE) is a tool to visualize high-dimensional data. It converts similarities between data points to joint probabilities and tries to minimize the Kullback-Leibler divergence between the joint probabilities of the low-dimensional embedding and the high-dimensional data. For more details on t-SNE including other use cases see this excellent *Toward Data Science* [blog post](https://towardsdatascience.com/an-introduction-to-t-sne-with-python-example-5a3a293108d1).Informally, t-SNE is preserving the local neighbourhood of data points to help uncover the manifold on which the data lies.  For example, a flat piece of paper with two coloured (e.g., red and blue) regions would be a simple manifold to characterize in 3D space; but, if the paper is crumpled up, it becomes very hard to characterize in the original 3D space (blue and red regions could be very close in this representational space) – instead, by following the cumpled paper (manifold) we would recover the fact that blue and red regions are really very distant, and not nearby at all.It is highly recommended to use another dimensionality reduction method (e.g. PCA) to reduce the number of dimensions to a reasonable amount if the number of features is very high. This will suppress some noise and speed up the computation of pairwise distances between samples.* [An Introduction to t-SNE with Python Example](https://towardsdatascience.com/an-introduction-to-t-sne-with-python-example-5a3a293108d1)<jupyter_code>import sklearn
import sklearn.decomposition
import sklearn.pipeline
import sklearn.manifold

_prng = np.random.RandomState(42)

embedding2_pipeline = sklearn.pipeline.make_pipeline(
    sklearn.decomposition.PCA(n_components=0.95, random_state=_prng),
    sklearn.manifold.TSNE(n_components=2, random_state=_prng))

embedding3_pipeline = sklearn.pipeline.make_pipeline(
    sklearn.decomposition.PCA(n_components=0.95, random_state=_prng),
    sklearn.manifold.TSNE(n_components=3, random_state=_prng))
# Sample 10% of the data

_prng = np.random.RandomState(42)

_idxs = _prng.randint(y_train_flat.shape[0], size=y_train_flat.shape[0] // 10)
tsne_features = x_train_flat[_idxs]
tsne_category = y_train_flat[_idxs]
print("tsne_features:", tsne_features.shape, 
      "tsne_category", tsne_category.shape, 
      "min,max category:", tsne_category.min(), tsne_category.max())
# t-SNE is SLOW (but can be GPU accelerated!); 
#       lengthy operation, be prepared to wait...

transform2_tsne_features = embedding2_pipeline.fit_transform(tsne_features)

print("transform2_tsne_features:", transform2_tsne_features.shape)
for i in range(2):
    print("min,max features[%s]:" % i, 
          transform2_tsne_features[:,i].min(), 
          transform2_tsne_features[:,i].max())
category_scatter_plot(transform2_tsne_features, tsne_category, title='CIFAR10 - t-SNE')<jupyter_output><empty_output><jupyter_text>**Note:** Skip this step during the tutorial, it will take too long to complete.<jupyter_code># t-SNE is SLOW (but can be GPU accelerated!); 
#       extremely lengthy operation, be prepared to wait... and wait...

transform3_tsne_features = embedding3_pipeline.fit_transform(tsne_features)

print("transform3_tsne_features:", transform3_tsne_features.shape)
for i in range(3):
    print("min,max features[%s]:" % i, 
          transform3_tsne_features[:,i].min(), 
          transform3_tsne_features[:,i].max())
category_scatter3d_plot(transform3_tsne_features, tsne_category, title='CIFAR10 - t-SNE')<jupyter_output><empty_output><jupyter_text>t-SNE relates the data points (images) according to their closest neighbours.  Hints of underlying categories appear; but are not cleanly seperable into the original categories.  Data ConversionThe data type for the training data is `uint8`, while the input type for the network will be `float32` so the data must be converted.  Also, the labels need to be categorical, or *one-hot encoded*, as discussed previously.  Keras provides utility functions to convert labels to categories (`to_categorical`), and `numpy` makes it easy to perform operations over entire arrays.* https://keras.io/examples/cifar10_cnn/<jupyter_code>num_classes = (y_train.max() - y_train.min()) + 1
print('num_classes =', num_classes)

y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255

train_data = (x_train, y_train)
test_data = (x_test, y_test)<jupyter_output><empty_output><jupyter_text>After the data conversion, notice that the datatypes are `float32`, the input `x` data shapes are the same; but, the `y` classification labels are now 10-dimensional, instead of scalar.<jupyter_code>print('x_train type:', type(x_train))
print('x_train dtype:', x_train.dtype)
print('x_train shape:', x_train.shape)
print('x_test shape:', x_test.shape)

print('y_train type:', type(y_train))
print('y_train dtype:', y_train.dtype)
print('y_train shape:', y_train.shape)
print('y_test shape:', y_test.shape)<jupyter_output><empty_output><jupyter_text>Acquire Pre-Trained NetworkDownload an *ImageNet* pretrained VGG16 network[1](fn1), sans classification layer, shaped for 32x32px colour images[*](https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5) (the smallest supported size).  This image-feature detection network is an example of a deep CNN (Convolutional Neural Network).**Note:** The network must be fixed – it was already trained on a very large dataset, so training it on our smaller dataset would result in it un-learning valuable generic features.[1] *Very Deep Convolutional Networks for Large-Scale Image Recognition** by <NAME> and <NAME>, [arXiv (2014)](https://arxiv.org/abs/1409.1556).<jupyter_code>cache_models()
from tensorflow.keras.applications import VGG16

conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(32, 32, 3))
conv_base.trainable = False
keras.utils.plot_model(conv_base)
conv_base.summary()<jupyter_output><empty_output><jupyter_text>The summary shows the layers, starting from the InputLayer and proceeding through Conv2D convolutional layers, which are then collected at MaxPooling2D layers.A convolutional kernel is a small matrix that looks for a specific, localized, pattern on its inputs.  This pattern is called a `feature`.  The kernel is applied at each location on the input image, and the output is another image – a feature image – that represent the strength of that feature at the given location.  Because the inputs to convolution are images, and the outputs are also images – but transformed into a different feature space – it is possible to stack many convolutional layers on top of each other.A feature image can be reduced in size with a MaxPooling2D layer.  This layer 'pools' an `MxN` region to a single value, taking the largest value from the 'pool'.  The 'Max' in 'MaxPooling' is keeping the *best* evidence for that feature, found in the original region.The InputLayer shape and data type should match with the input data:*Note:* The first dimension of the shape will differ; the input layer has `None` to indicate it accepts a batch sized collection of arrays of the remaining shape.  The input data shape will indicate, in that first axis, how many samples it contains.<jupyter_code>print("input layer shape:", conv_base.layers[0].input.shape)
print("input layer dtype:", conv_base.layers[0].input.dtype)
print("input layer type:", type(conv_base.layers[0].input))
print("input data shape:", x_train.shape)
print("input data dtype:", x_train.dtype)
print("input data type:", type(x_train))<jupyter_output><empty_output><jupyter_text>Explore Convolutional LayersThe following are visualization functions (and helpers) for understanding what the convolutional layers in a network have learned.We may ask questions about each convolutional kernal in a convolutional layer:* What local features is the kernel looking for: `visualize_conv_layer_weights`* For a given input image, what feature image will the kernal produce: `visualize_conv_layer_output`* What input image makes the kernel respond most strongly: `visualize_conv_layer_response`<jupyter_code>def cifar10_image_plot(img_data=None, image_index=None):
    (x_imgs, y_imgs) = img_data if img_data else (x_train, y_train)

    if not image_index:
        image_index = int(random.uniform(0, x_imgs.shape[0]))

    plt.imshow(x_imgs[image_index], cmap='gray')
    plt.title("%s" % cifar10_label(y_imgs[image_index]))
    plt.xlabel("#%s" % image_index)
    plt.show()
    
    return image_index

def get_model_layer(model, layer_name):
    if type(layer_name) == str:
        layer = model.get_layer(layer_name)
    else:
        m = model
        for ln in layer_name:
            model = m
            m = m.get_layer(ln)
        layer = m
    return (model, layer)
def visualize_conv_layer_weights(model, layer_name):
    (model, layer) = get_model_layer(model, layer_name)
    layer_weights = layer.weights[0]

    max_size = layer_weights.shape[3]
    col_size = 12
    row_size = int(np.ceil(float(max_size) / float(col_size)))

    print("conv layer: %s shape: %s size: (%s,%s) count: %s" % 
          (layer_name,
           layer_weights.shape,
           layer_weights.shape[0], layer_weights.shape[1],
           max_size))

    fig, ax = plt.subplots(row_size, col_size, figsize=(12, 1.2 * row_size))
    idx = 0

    for row in range(0,row_size):
        for col in range(0,col_size):
            ax[row][col].set_xticks([])
            ax[row][col].set_yticks([])
            if idx < max_size:
                ax[row][col].imshow(layer_weights[:, :, 0, idx], cmap='gray')
            else:
                fig.delaxes(ax[row][col])
            idx += 1

    plt.tight_layout()
    plt.show()
def visualize_conv_layer_output(model, layer_name, image_index=None):
    (model, layer) = get_model_layer(model, layer_name)
    layer_output = layer.output

    if not image_index:
        image_index = cifar10_image_plot()
        
    intermediate_model = keras.models.Model(inputs = model.input, outputs=layer_output) 
    intermediate_prediction = intermediate_model.predict(x_train[image_index].reshape(1,32,32,3))
  
    max_size = layer_output.shape[3]
    col_size = 10
    row_size = int(np.ceil(float(max_size) / float(col_size)))

    print("conv layer: %s shape: %s size: (%s,%s) count: %s" % 
          (layer_name,
           layer_output.shape,
           layer_output.shape[1], layer_output.shape[2],
           max_size))
    
    fig, ax = plt.subplots(row_size, col_size, figsize=(12, 1.2 * row_size))
    idx = 0

    for row in range(0,row_size):
        for col in range(0,col_size):
            ax[row][col].set_xticks([])
            ax[row][col].set_yticks([])
            if idx < max_size:
                ax[row][col].imshow(intermediate_prediction[0, :, :, idx], cmap='gray')
            else:
                fig.delaxes(ax[row][col])
            idx += 1

    plt.tight_layout()
    plt.show()
from tensorflow.keras import backend as K

def process_image(x):
    epsilon = 1e-5
    # Normalizes the tensor: centers on 0, ensures that std is 0.1 Clips to [0, 1]
    x -= x.mean()
    x /= (x.std() + epsilon)
    x *= 0.1
    x += 0.5
    x = np.clip(x, 0, 1)
    x *= 255
    x = np.clip(x, 0, 255).astype('uint8')
    return x

def generate_response_pattern(model, conv_layer_output, filter_index=0):
    #step_size = 1.0
    epsilon = 1e-5

    img_tensor = tf.Variable(tf.random.uniform((1, 32, 32, 3)) * 20 + 128.0, trainable=True)

    response_model = keras.models.Model([model.inputs], [conv_layer_output])

    for i in range(40):
        with tf.GradientTape() as gtape:
            layer_output = response_model(img_tensor)
            loss = K.mean(layer_output[0, :, :, filter_index])
            grads = gtape.gradient(loss, img_tensor)
            grads /= (K.sqrt(K.mean(K.square(grads))) + epsilon)
        img_tensor = tf.Variable(tf.add(img_tensor, grads))

    img = np.array(img_tensor[0])
    return process_image(img)
def visualize_conv_layer_response(model, layer_name):
    (model, layer) = get_model_layer(model, layer_name)
    layer_output = layer.output
    
    max_size = layer_output.shape[3]
    col_size = 12
    row_size = int(np.ceil(float(max_size) / float(col_size)))

    print("conv layer: %s shape: %s size: (%s,%s) count: %s" % 
          (layer_name,
           layer_output.shape,
           layer_output.shape[1], layer_output.shape[2],
           max_size))
    
    fig, ax = plt.subplots(row_size, col_size, figsize=(12, 1.2 * row_size))
    idx = 0

    for row in range(0,row_size):
        for col in range(0,col_size):
            ax[row][col].set_xticks([])
            ax[row][col].set_yticks([])
            if idx < max_size:
                img = generate_response_pattern(model, layer_output, idx)
                ax[row][col].imshow(img, cmap='gray')
                ax[row][col].set_title("%s" % idx)
            else:
                fig.delaxes(ax[row][col])
            idx += 1

    plt.tight_layout()
    plt.show()<jupyter_output><empty_output><jupyter_text>Looking at the the first 4 convolution layers, we see that:* All the kernels are 3x3 (i.e., 9 elements each)* Layers 1 & 2 have 64 kernels each (64 different possible features)* Layers 3 & 4 have 128 kernels each (128 different possible features)* Light pixels indicate preference for an activated pixel* Dark pixels indicate preference for an inactive pixel* The kernels seem to represent edges and lines at various angles<jupyter_code>for n in [l.name for l in conv_base.layers if isinstance(l, keras.layers.Conv2D)][:4]:
    visualize_conv_layer_weights(conv_base, n)<jupyter_output><empty_output><jupyter_text>For the given input image, show the corresponding feature image. At the lower level layers (e.g., first Conv2D layer), the feature images seem to capture concepts like 'edges' or maybe 'solid colour'?At higher layers, the size of the feature images decrease because of the MaxPooling.  They also appear more abstract – harder to visually recognize than the original image – however, the features are spatially related to the original image (e.g., if there is a white/high value in the lower-left corner of the feature image, then somewhere on the lower-left corner of the original image, there exists pixels that the network is confident represent the feature in question).<jupyter_code>image_index = cifar10_image_plot()
for n in [l.name for l in conv_base.layers if isinstance(l, keras.layers.Conv2D)][:7]:
    visualize_conv_layer_output(conv_base, n, image_index)<jupyter_output><empty_output><jupyter_text>This plot shows which input images cause the greatest response from the convolution kernels.  At lower layers, we see many simple 'wave' textures showing that these kernals like to see edges at particular angles.  At lower-middle layers, the paterns show larger scale and more complexity (like dots and curves); but, still lots of angled edges.<jupyter_code>for n in [l.name for l in conv_base.layers if isinstance(l, keras.layers.Conv2D)][:4]:
    visualize_conv_layer_response(conv_base, n)<jupyter_output><empty_output><jupyter_text>The patterns in the higher levels can get even more complex; but, some of them don't seem to encode for anything but noise.  Maybe these could be pruned to make a smaller network...**Note:** Skip this step during the tutorial, it will take too long to complete.<jupyter_code># NOTE: Visualize mid to higher level convolutional layers; 
#       lengthy operation, be prepared to wait...
for n in [l.name for l in conv_base.layers if isinstance(l, keras.layers.Conv2D)][4:]:
    visualize_conv_layer_response(conv_base, n)<jupyter_output><empty_output><jupyter_text>CNN Base + Classifier ModelCreate a simple model that has the pre-trained CNN (Convolutional Neural Network) as a base, and adds a basic classifier on top.The new layer types are Flatten, Dense, Dropout, and Activation.The Flatten layer reshapes the input dimensions (2D + 1 channel) into a single dimension.The Dense(x) layer is a layer of (`x`) neurons (represented as a flat 1D array) connected to a flat input.  The size of the input and outputs do not need to match.The Dropout(x) layer withholds a random fraction (`x`) of the input neurons from training during each batch of data.  This limits the ability of the network to `overfit` on the training data (i.e., memorize training data, rather than  learn generalizable rules).Activation is an essential part of (or addition to) each layer.  Layers like Dense are simply linear functions (weighted sums + a bias).  Without a non-linear component, the network could not learn a non-linear function.  Activations like 'relu' (Rectified Linear Unit), 'tanh', or 'sigmoid' are functions to introduce a non-linearity.  They also clamp output values within known ranges.The 'softmax' activation is used to produce probability distributions over multiple categories.This example uses the Sequential API to build the final network.* [Activation Functions in Neural Networks](https://towardsdatascience.com/activation-functions-neural-networks-1cbd9f8d91d6)<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense, Activation, Dropout
from tensorflow.keras.applications import VGG16

def create_cnnbase_classifier_model(conv_base=None):
    if not conv_base:
        conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(32, 32, 3))
        conv_base.trainable = False

    model = Sequential()
    model.add(conv_base)
    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes))
    model.add(Activation('softmax'))

    return model<jupyter_output><empty_output><jupyter_text>Create our model *model_transfer_cnn* by calling the creation function *create_cnnbase_classifier_model* above.Notice the split of total parameters (\~15 million) between trainable (\~0.3 million for our classifier) and non-trainable (\~14.7 million for the pre-trained CNN).Note also that the final Dense layer squeezes the network down to the number of categories.<jupyter_code>model_transfer_cnn = create_cnnbase_classifier_model(conv_base)
model_transfer_cnn.summary()<jupyter_output><empty_output><jupyter_text>Train ModelTraining a model typically involves setting relevant hyperparameters that control aspects of the training process.  Common hyperparameters include:* `epochs`: The number of training passes through the entire dataset.  The number of epochs depends upon the complexity of the dataset, and how effectively the network architecture of the model can learn it.  If the value is too small, the model accuracy will be low.  If the value is too big, then the training will take too long for no additional benefit, as the model accuracy will plateau.* `batch_size`: The number of samples to train during each step. The number should be set so that the GPU memory and compute are well utilized.  The `learning_rate` needs to be set accordingly.* `learning_rate`:  The step-size to update model weights during the training update phase (backpropagation).  Too small, and learning takes too long.  Too large, and we may step over the minima we are trying to find.  The learning rate can be increased as the batch sizes increases (with some caveats), on the assumption that with more data in a larger batch, the error gradient will be more accurate, so therefore, we can take a larger step.* `decay`: Used by some optimizers to decrease the `learning_rate` over time, on the assumption that as we get closer to our goal, we should focus on smaller refinement steps.<jupyter_code>batch_size = 128 #32
epochs = 25 #100
learning_rate = 1e-3 #1e-4
decay = 1e-6<jupyter_output><empty_output><jupyter_text>The model needs to be compiled prior to use.  This step enables the model to train efficiently on the GPU device.This step also specifies the loss functions, accuracy metrics, learning strategy (optimizers), and more.Our `loss` is *categorical_crossentropy* because we are doing multi-category classification.We use an RMSprop optimizer, which is a varient of standard gradient descent optimizers that also includes momentum.  Momentum is used to speed up learning in directions where it has been making more progress.* [A Look at Gradient Descent and RMSprop Optimizers](https://towardsdatascience.com/a-look-at-gradient-descent-and-rmsprop-optimizers-f77d483ef08b)* [Understanding RMSprop — faster neural network learning](https://towardsdatascience.com/understanding-rmsprop-faster-neural-network-learning-62e116fcf29a)<jupyter_code>from tensorflow.keras.optimizers import RMSprop

model_transfer_cnn.compile(loss='categorical_crossentropy',
                           optimizer=RMSprop(learning_rate=learning_rate, decay=decay),
                           metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>The model `fit` function trains the network, and returns a history of training and testing accuracy.*Note:* Because we already have a test dataset, and we are not validating our hyperparameters, we will use the test dataset for validation.  We could have also reserved a fraction of the training data to use for validation.<jupyter_code>%%time
history = model_transfer_cnn.fit(x_train, y_train,
                                 batch_size=batch_size,
                                 epochs=epochs,
                                 validation_data=(x_test, y_test),
                                 shuffle=True)<jupyter_output><empty_output><jupyter_text>Evaluate Model Visualize accuracy and loss for training and validation.* https://keras.io/visualization/<jupyter_code>def history_plot(history):
    fig = plt.figure(figsize=(12,5))

    plt.title('Model accuracy & loss')

    # Plot training & validation accuracy values
    ax1 = fig.add_subplot()
    #ax1.set_ylim(0, 1.1 * max(history.history['loss']+history.history['val_loss']))
    ax1.set_prop_cycle(color=['green', 'red'])
    p1 = ax1.plot(history.history['loss'], label='Train Loss')
    p2 = ax1.plot(history.history['val_loss'], label='Test Loss')

    # Plot training & validation loss values
    ax2 = ax1.twinx()
    ax2.set_ylim(0, 1.1 * max(history.history['accuracy']+history.history['val_accuracy']))
    ax2.set_prop_cycle(color=['blue', 'orange'])
    p3 = ax2.plot(history.history['accuracy'], label='Train Acc')
    p4 = ax2.plot(history.history['val_accuracy'], label='Test Acc')

    ax1.set_ylabel('Loss')
    ax1.set_xlabel('Epoch')
    ax2.set_ylabel('Accuracy')

    pz = p3 + p4 + p1 + p2
    plt.legend(pz, [l.get_label() for l in pz], loc='center right')
    plt.show()<jupyter_output><empty_output><jupyter_text>The history plot shows characteristic features of training performance over successive epochs.  Accuracy and loss are related, in that a reduction in loss produces an increase in accuracy.  The graph shows characteristic arcs for training and testing accuracy / loss over training time (epochs).The primary measure to improve is *testing accuracy*, because that indicates how well the model generalizes to data it must typically classify.The accuracy curves show that testing accuracy has plateaued (with some variability), while training accuracy increases (but at a slowing rate).  The difference between training and testing accuracy shows overfitting of the model (i.e., the model can memorize what it has seen better than it can generalize the classification rules).We would like a model that *can* overfit (otherwise it might not be large enough to capture the complexity of the data domain), but doesn't.  And then, it is only trained until *test accuracy* peaks.Could the model 100% overfit the data?  The graph doesn't answer definitively yet, but training accuracy seems to be slowing, while training loss is still decreasing (with lots of room to improve – the loss axis does not start at zero).*Note:* The model contains Dropout layers to help prevent overfitting.  What happens to training and testing accuracy when those layers are removed?<jupyter_code>history_plot(history)
# Score trained model.
scores = model_transfer_cnn.evaluate(x_test, y_test, verbose=0)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])<jupyter_output><empty_output><jupyter_text>The following prediction plot functions provide insight into aspects of model prediction.<jupyter_code>def prediction_plot(model, test_data):
    (x_test, y_test) = test_data
    fig = plt.figure(figsize=(16,8))
    correct = 0
    total = 0
    rSym = ''
    
    for i in range(40):
        plt.subplot(4, 10, i + 1)
        plt.xticks([])
        plt.yticks([])
        idx = int(random.uniform(0, x_test.shape[0]))
        result = model.predict(x_test[idx:idx+1])[0]
        if y_test is not None:
            rCorrect = True if cifar10_label(y_test[idx]) == cifar10_label(result) else False
            rSym = '✔' if rCorrect else '✘'
            correct += 1 if rCorrect else 0
        total += 1
        plt.title("%s %s" % (rSym, cifar10_label(result)))
        plt.imshow(x_test[idx], cmap=plt.get_cmap('gray'))
    plt.show()
    
    if y_test is not None:
        print("% 3.2f%% correct (%s/%s)" % (100.0 * float(correct) / float(total), correct, total))
def prediction_classes_plot(model, test_data):
    (x_test, y_test) = test_data
    fig = plt.figure(figsize=(16,8))
    correct = 0
    total = 0
    rSym = ''
    
    for i in range(40):
        plt.subplot(4, 10, i + 1)
        plt.xticks([])
        plt.yticks([])
        idx = int(random.uniform(0, x_test.shape[0]))
        result = model.predict_classes(x_test[idx:idx+1])[0]
        if y_test is not None:
            rCorrect = True if cifar10_label(y_test[idx]) == cifar10_label(result) else False
            rSym = '✔' if rCorrect else '✘'
            correct += 1 if rCorrect else 0
        total += 1
        plt.title("%s %s" % (rSym, cifar10_label(result)))
        plt.imshow(x_test[idx], cmap=plt.get_cmap('gray'))
    plt.show()
    
    if y_test is not None:
        print("% 3.2f%% correct (%s/%s)" % (100.0 * float(correct) / float(total), correct, total))
def prediction_proba_plot(model, test_data):
    (x_test, y_test) = test_data
    fig = plt.figure(figsize=(15,15))
    
    for i in range(10):
        plt.subplot(10, 2, (2*i) + 1)
        plt.xticks([])
        plt.yticks([])
        idx = int(random.uniform(0, x_test.shape[0]))
        result = model.predict_proba(x_test[idx:idx+1])[0] * 100 # prob -> percent
        if y_test is not None:
            plt.title("%s" % cifar10_label(y_test[idx]))
        plt.xlabel("#%s" % idx)
        plt.imshow(x_test[idx], cmap=plt.get_cmap('gray'))
        
        ax = plt.subplot(10, 2, (2*i) + 2)
        plt.bar(np.arange(len(result)), result, label='%')
        plt.xticks(range(0, len(result) + 1))
        ax.set_xticklabels(cifar10_label_names)
        plt.title("classifier probabilities")

        plt.tight_layout()
    plt.show()<jupyter_output><empty_output><jupyter_text>* *Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization* by <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, and <NAME> [arXiv (2016)](https://arxiv.org/abs/1610.02391)* https://jacobgil.github.io/deeplearning/class-activation-maps* https://keras.io/examples/vision/grad_cam/<jupyter_code>from tensorflow.keras import backend as K

def generate_activation_pattern(model, conv_layer_output, category_idx, image):
    epsilon = 1e-10

    activation_model = keras.models.Model([model.inputs], [conv_layer_output, model.output])

    with tf.GradientTape() as gtape:
        conv_output, prediction = activation_model(image)
        category_output = prediction[:, category_idx]
        grads = gtape.gradient(category_output, conv_output)
        pooled_grads = K.mean(grads, axis=(0, 1, 2))

    heatmap = tf.reduce_mean(tf.multiply(pooled_grads, conv_output), axis=-1) * -1.
    heatmap = np.maximum(heatmap, 0)
    heatmap /= np.max(heatmap) + epsilon
    return(heatmap)
def activation_plot(model, layer_name, image_data, image_index=None):
    (layer_model, conv_layer) = get_model_layer(model, layer_name)
    (x_imgs, y_cat) = image_data

    if not image_index:
        image_index = int(random.uniform(0, x_imgs.shape[0]))
    
    image = x_imgs[image_index:image_index+1]

    fig = plt.figure(figsize=(16,8))

    plt.subplot(1, num_classes + 2, 1)
    plt.xticks([])
    plt.yticks([])
    plt.title(cifar10_label(y_cat[image_index]))
    plt.xlabel("#%s" % image_index)
    plt.imshow(image.reshape(32, 32, 3))

    result = model.predict(image)[0]

    for i in range(num_classes):
        activation = generate_activation_pattern(model, conv_layer.output, i, image)
        activation = np.copy(activation)
        plt.subplot(1, num_classes + 2, i + 2)
        plt.xticks([])
        plt.yticks([])
        plt.title(cifar10_label(i))
        plt.xlabel("(% 3.2f%%)" % (result[i] * 100.0))
        plt.imshow(activation[0])
        
    plt.show()<jupyter_output><empty_output><jupyter_text>This plot shows what the model thinks is the most likely class for each image.<jupyter_code>prediction_classes_plot(model_transfer_cnn, (x_test, y_test))<jupyter_output><empty_output><jupyter_text>This plot shows the probabilities that the model assigns to each category class, and provides a sense of how confident the network is with its classifications.<jupyter_code>prediction_proba_plot(model_transfer_cnn, (x_test, y_test))
# TODO: Complete activation plot
#activation_plot(model_transfer_cnn, ('vgg16', 'block5_conv3'), (x_test, y_test), 1)<jupyter_output><empty_output><jupyter_text>CNN Classifier ModelCreate a basic CNN (Convolutional Neural Network) based classifier from scratch.We have encountered Conv2D and MaxPooling2D layers previously, but here we see how they are declared.  Conv2D layers specify the number of convolution kernels and their shape.  MaxPooling2D layers specify the size of each pool (i.e., the scaling factors).Notice the total number of parameters (\~1.25 million) in this smaller network.<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Flatten, Dense, Activation, Dropout, Conv2D, MaxPooling2D

def create_cnn_classifier_model():
    model = Sequential()
    model.add(Conv2D(32, (3, 3), padding='same',
                     input_shape=x_train.shape[1:]))
    model.add(Activation('relu'))
    model.add(Conv2D(32, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Conv2D(64, (3, 3), padding='same'))
    model.add(Activation('relu'))
    model.add(Conv2D(64, (3, 3)))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))

    model.add(Flatten())
    model.add(Dense(512))
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(num_classes))
    model.add(Activation('softmax'))

    return model
model_simple_cnn = create_cnn_classifier_model()
model_simple_cnn.summary()
batch_size = 128 #32
epochs = 25 #100
learning_rate = 1e-3 #1e-4
decay = 1e-6
from tensorflow.keras.optimizers import RMSprop

model_simple_cnn.compile(loss='categorical_crossentropy',
                         optimizer=RMSprop(learning_rate=learning_rate, decay=decay),
                         metrics=['accuracy'])
%%time
history = model_simple_cnn.fit(x_train, y_train,
                               batch_size=batch_size,
                               epochs=epochs,
                               validation_data=(x_test, y_test),
                               shuffle=True)<jupyter_output><empty_output><jupyter_text>The notable features of the history plot for this model are:* Training accuracy is ~10 percentage points better than the previous model,* test accuracy more closely tracks training accuracy, and* test accuracy shows more variability.<jupyter_code>history_plot(history)
# Score trained model.
scores = model_simple_cnn.evaluate(x_test, y_test, verbose=0)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])
prediction_classes_plot(model_simple_cnn, (x_test, y_test))
prediction_proba_plot(model_simple_cnn, (x_test, y_test))
for n in [l.name for l in model_simple_cnn.layers if isinstance(l, keras.layers.Conv2D)][:4]:
    visualize_conv_layer_weights(model_simple_cnn, n)
image_index = cifar10_image_plot()
for n in [l.name for l in model_simple_cnn.layers if isinstance(l, keras.layers.Conv2D)]:
    visualize_conv_layer_output(model_simple_cnn, n, image_index)<jupyter_output><empty_output><jupyter_text>Interesting aspects of the convolutional layer response for our *model_simple_cnn* model:* There are fewer Conv2D layers in this simple model* Compared to the pre-trained VGG16 convolutional base network,   * the latter levels are the first edge detection kernels, and   * there are no layers with higher-level features.<jupyter_code>for n in [l.name for l in model_simple_cnn.layers if isinstance(l, keras.layers.Conv2D)][:4]:
    visualize_conv_layer_response(model_simple_cnn, n)<jupyter_output><empty_output><jupyter_text>This plot shows which pixels of the original image contributed the most 'confidence' to the classification categories.The technique is better applied to larger images where the object of interest might be anywhere inside the image.<jupyter_code>n = [l.name for l in model_simple_cnn.layers if isinstance(l, keras.layers.Conv2D)][-1]
print(n)
for i in range(5):
    activation_plot(model_simple_cnn, n, (x_test, y_test))<jupyter_output><empty_output><jupyter_text>Combined ModelsKeras supports a functional interface to take network architectures beyond simply sequential networks.The new layer types are Input and Concatenate; and, there is an explicit Model class.The Input layer is a special layer denoting sources of input from training batches.The Concatenate layer combines multiple inputs (along an axis with the same size) and creates a larger layer incorporating all the input values.Model construction is also different.  Instead of using a `Sequential` model, and `add`ing layers to it:* An explicit Input layer is created, * we pass inputs into the layers explicity,* the output from a layer become input for arbitrary other layers, and finally,* A Model object is created with the source Input layer as inputs and outputs from the final layer.We'll demonstrate by creating a new network which combines the two CNN classifier networks we created previously.*Note:* Network models provided as an argument are changed to be non-trainable (the assumption is that they were already trained).<jupyter_code>from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Concatenate, Flatten, Dense, Activation, Dropout
from tensorflow.keras.optimizers import RMSprop

def create_combined_classifier_model(trained_model1=None, trained_model2=None):
    if trained_model1:
        network1 = trained_model1
        network1.trainable = False
    else:
        network1 = create_cnnbase_classifier_model()

    if trained_model2:
        network2 = trained_model2
        network2.trainable = False
    else:
        network2 = create_cnn_classifier_model()

    inputs = Input(shape=(32,32,3), name='cifar10_image')
    c1 = network1(inputs)
    c2 = network2(inputs)
    c = Concatenate()([c1, c2])
    x = Dense(512)(c)
    x = Activation('relu')(x)
    x = Dropout(0.5)(x)
    x = Dense(num_classes)(x)
    outputs = Activation('softmax')(x)
    
    model = Model(inputs=inputs, outputs=outputs, name='combined_cnn_classifier')
    return model<jupyter_output><empty_output><jupyter_text>Combining Pre-Trained ModelsThis version of the combined classifier uses both of the trained networks we created previously.Notice the trainable parameters (~16,000) is very small.  How will this affect training?<jupyter_code>model_combined = create_combined_classifier_model(model_transfer_cnn, model_simple_cnn)
model_combined.summary()<jupyter_output><empty_output><jupyter_text>This plot shows a graph representation of the layer connections.  Notice how a single input feeds the previously created Sequential networks, their output is combine via Concatenate, and then a classifier network is added on top.<jupyter_code>keras.utils.plot_model(model_combined)<jupyter_output><empty_output><jupyter_text>Reduce number of `epochs` because this network is mostly trained (execpt for the final classifier), and there are few trainable parameters.<jupyter_code>batch_size = 128 #32
epochs = 5 #100
learning_rate = 1e-3 #1e-4
decay = 1e-6

model_combined.compile(loss='categorical_crossentropy',
                         optimizer=RMSprop(learning_rate=learning_rate, decay=decay),
                         metrics=['accuracy'])
%%time
history = model_combined.fit(x_train, y_train,
                             batch_size=batch_size,
                             epochs=epochs,
                             validation_data=(x_test, y_test),
                             shuffle=True)<jupyter_output><empty_output><jupyter_text>It looks like everything we needed to learn was learned in a single epoch.<jupyter_code>history_plot(history)<jupyter_output><empty_output><jupyter_text>Here is an interesting, possibly counter-intuitive, result: combining two weaker networks can create a stronger one.The reason is that the weakness in one model, might be a strength in the other model (each has 'knowledge' that the other doesn't); we just need a layer to discriminate when to trust each model.  At a larger scale (of layers and models) is what is happening at the lower level of the neurons themselves.<jupyter_code># Score trained model.
scores = model_combined.evaluate(x_test, y_test, verbose=0)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])
# NOTE: Sequential Model provides `predict_classes` or `predict_proba`
#       Functional API Model does not; because it may have multiple outputs
# Using simple `predict` plot instead
prediction_plot(model_combined, (x_test, y_test))<jupyter_output><empty_output><jupyter_text>The combine model improves accuracy by 0.5-2% (with TF 2.0), and takes 1/5th of the time to train. Training Combining ModelsThis version of the combined classifier uses both network architectures seen previously; except, in this version, the models need to be trained from scratch.  The following cells repeat the previous experiments with this combined classifier.*Spoiler:* The combined network doesn't perform any better than the partially trained one did, but takes much longer to train (more epochs).**Note:** Skip this step during the tutorial, it will cause unecessary delay.<jupyter_code>%%time
batch_size = 128 #32
epochs = 25 #100
learning_rate = 1e-3 #1e-4
decay = 1e-6

model_combined = create_combined_classifier_model()
model_combined.compile(loss='categorical_crossentropy',
                         optimizer=RMSprop(learning_rate=learning_rate, decay=decay),
                         metrics=['accuracy'])
history = model_combined.fit(x_train, y_train,
                             batch_size=batch_size,
                             epochs=epochs,
                             validation_data=(x_test, y_test),
                             shuffle=True)
history_plot(history)
# Score trained model.
scores = model_combined.evaluate(x_test, y_test, verbose=0)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])<jupyter_output><empty_output><jupyter_text>Skip ConnectionsFrom previous comparisons of the `visualize_conv_layer_response` plots of the two basic CNN models, it becomes apparent that the pre-trained VGG16 network contains more complex *knowledge* about images: there were more convolutional layers with a greater variety of patterns and features they could represent.In the previous cnnbase_classifier model `model_transfer_cnn`, only the last Conv2D layer fed directly to the classifier, and the feature information contained in the middle layers wasn't directly available to the classifier.Skip Connections are a way to bring lower level feature encodings to higher levels of the network directly.  They are also useful during training very deep networks to deal with the problem of *vanishing gradients*.In the following example, the original CNN base of the pre-trained VGG16 model is decomposed into layered groups, and a new network created that feeds these intermediate layers to the top of the network, where they are concatenated together to perform the final classification.* https://towardsdatascience.com/understanding-and-coding-a-resnet-in-keras-446d7ff84d33* https://arxiv.org/abs/1608.04117<jupyter_code>from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Input, Concatenate, Flatten, Dense, Activation, Dropout
from tensorflow.keras.applications import VGG16
from tensorflow.keras.optimizers import RMSprop

def create_cnnbase_skipconnected_classifier_model(conv_base=None):
    if not conv_base:
        conv_base = VGG16(weights='imagenet', include_top=False, input_shape=(32, 32, 3))
        conv_base.trainable = False

    # Split conv_base into groups of CNN layers topped by a MaxPooling2D layer
    cb_idxs = [i for (i,l) in enumerate(conv_base.layers) if isinstance(l, keras.layers.MaxPooling2D)]
    all_idxs = [-1] + cb_idxs
    idx_pairs = [l for l in zip(all_idxs, cb_idxs)]
    cb_layers = [conv_base.layers[i+1:j+1] for (i,j) in idx_pairs]

    # Dense Pre-Classifier Layers creation function - used repeatedly at multiple network locations
    def dense_classes(l):
        x = Dense(512)(l)
        x = Activation('relu')(x)
        x = Dropout(0.5)(x)
        x = Dense(num_classes)(x)
        return x
    
    inputs = Input(shape=(32,32,3), name='cifar10_image')

    # Join split groups into a sequence, but keep track of their outputs to create skip connections
    skips = []
    inz = inputs
    for lz in cb_layers:
        m = Sequential()
        m.trainable = False
        for ls in lz:
            m.add(ls)
        # inz is the output of model m, but the input for next layer group
        inz = m(inz)
        skips += [inz]

    # Flatten all outputs (which had different dimensions) to Concatenate them on a common axis
    flats = [dense_classes(Flatten()(l)) for l in skips]
    c = Concatenate()(flats)
    x = dense_classes(c)
    outputs = Activation('softmax')(x)

    model = Model(inputs=inputs, outputs=outputs)
    
    return model
model_skipconnected = create_cnnbase_skipconnected_classifier_model(conv_base)
model_skipconnected.summary()
keras.utils.plot_model(model_skipconnected)
%%time
batch_size = 128 #32
epochs = 25 #100
learning_rate = 1e-3 #1e-4
decay = 1e-6

model_skipconnected.compile(loss='categorical_crossentropy',
                         optimizer=RMSprop(learning_rate=learning_rate, decay=decay),
                         metrics=['accuracy'])
history = model_skipconnected.fit(x_train, y_train,
                                  batch_size=batch_size,
                                  epochs=epochs,
                                  validation_data=(x_test, y_test),
                                  shuffle=True)
history_plot(history)<jupyter_output><empty_output><jupyter_text>A significant improvement over the first pre-trained model.<jupyter_code># Score trained model.
scores = model_skipconnected.evaluate(x_test, y_test, verbose=0)
print('Test loss:', scores[0])
print('Test accuracy:', scores[1])
# Using simple `predict` plot because model uses Functional API
prediction_plot(model_skipconnected, (x_test, y_test))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09690194004031624, 'fraction_numerical': 0.014796917666233004, 'mean_word_length': 4.299492385786802, 'pattern_counts': {'":': 0, '<': 225, '<?xml version=': 0, '>': 229, 'https://': 26, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 27, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '52', 'dir': 'jupyter-structured-clean-dedup', 'id': '337459', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 21873, 'n_tokens_neox': 20870, 'n_words': 7130, 'path': 'notebooks/keras-transfer-learning-tutorial.ipynb', 'repo_name': 'kaust-vislab/keras-tutorials-lite', 'size': '70062'}