<commit_before>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "77gENRVX40S7"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "d8jyt37T42Vf"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "aPxHdjwW5P2j"
      },
      "outputs": [],
      "source": [
        "#@title MIT License\n",
        "#\n",
        "# Copyright (c) 2017 <NAME>                                                                                                                    # IGNORE_COPYRIGHT: cleared by OSS licensing\n",
        "#\n",
        "# Permission is hereby granted, free of charge, to any person obtaining a\n",
        "# copy of this software and associated documentation files (the \"Software\"),\n",
        "# to deal in the Software without restriction, including without limitation\n",
        "# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n",
        "# and/or sell copies of the Software, and to permit persons to whom the\n",
        "# Software is furnished to do so, subject to the following conditions:\n",
        "#\n",
        "# The above copyright notice and this permission notice shall be included in\n",
        "# all copies or substantial portions of the Software.\n",
        "#\n",
        "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n",
        "# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n",
        "# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n",
        "# DEALINGS IN THE SOFTWARE."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hRTa3Ee15WsJ"
      },
      "source": [
        "# Transfer Learning Using Pretrained ConvNets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dQHMcypT3vDT"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/images/transfer_learning\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/transfer_learning.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/images/transfer_learning.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2X4KyhORdSeO"
      },
      "source": [
        "In this tutorial we will discuss how to classify cats vs dogs images by using transfer learning from a pre-trained network. This will allows us to get higher accuracies than we saw by training our network from scratch.\n",
        "\n",
        "A **pre-trained model **is a saved network that was previously trained on a large dataset, typically on a large-scale image-classification task. We can either use the pretrained model as it is or transfer learning using the pretrained convents. The intuition behind **transfer learning **is that if this model trained on a large and general enough dataset, this model will effectively serve as a generic model of the visual world. We can leverage these learned feature maps without having to train a large model on a large dataset by using these models as the basis of our own model specific to our task. There are 2 scenarios of transfer learning using a pretrained model:\n",
        "\n",
        "1. **Feature Extraction** - use the representations of learned by a previous network to extract meaningful features from new samples. We simply add a new classifier, which will be trained from scratch, on top of the pretrained model so that we can repurpose the feature maps learned previously for our dataset. **Do we use the entire pretrained model or just the convolutional base?** - We use the feature extraction portion of these pretrained convnets (convolutional base) since they are likely to be generic features and learned concepts over a picture. However, the classification part of the pretrained model is often specific to original classification task, and subsequently specific to the set of classes on which the model was trained.\n",
        "2. **Fine-Tuning** - unfreezing a few of the top layers of a frozen model base used for feature extraction, and jointly training both the newly added classifier layers as well as the last layers of the frozen model. This allows us to \"fine tune\" the higher order feature representations in addition to our final classifier in order to make them more relevant for the specific task involved.\n",
        "\n",
        "**We will follow the general machine learning workflow:**\n",
        "1. Examine and understand data\n",
        "2. Build an input pipeline - using Keras ImageDataGenerator as we did in the image classification tutorial\n",
        "3. Compose our model\n",
        "  * Load in our pretrained model (and pretrained weights)\n",
        "  * Stack our classification layers on top\n",
        "4. Train our model\n",
        "5. Evaluate model\n",
        "\n",
        "We will see an example of using the pre-trained convnet as the feature extraction and then fine-tune to train the last few layers of the base model.\n",
        "\n",
        "**Audience:** This post is geared towards beginners with some Keras API and ML background. To get the most out of this post, you should have some basic ML background, know what CNNs are, and be familiar with the Keras Sequential API.\n",
        "\n",
        "**Time Estimated**: 30 minutes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "iBMcobPHdD8O"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "import os\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "print(\"TensorFlow version is \", tf.__version__)\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.image as mpimg"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "v77rlkCKW0IJ"
      },
      "source": [
        "## Data preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aXzwKdouXf1h"
      },
      "source": [
        "### Download data - cats_and_dogs_filtered.zip\n",
        "We will download a filtered version of Kaggle's [Dogs vs Cats](https://www.kaggle.com/c/dogs-vs-cats/data) dataset. Then store the downloaded zip file to the \"/tmp/\" directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "nRnO59Kr6enO"
      },
      "outputs": [],
      "source": [
        "zip_file = tf.keras.utils.get_file(origin=\"https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip\",\n",
        "                                   fname=\"cats_and_dogs_filtered.zip\", extract=True)\n",
        "base_dir, _ = os.path.splitext(zip_file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9_6h-c5EXN91"
      },
      "source": [
        "### Prepare training and validation cats and dogs datasets\n",
        "Create the training and validation directories for cats datasets and dog datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RWcldM4TXLen"
      },
      "outputs": [],
      "source": [
        "train_dir = os.path.join(base_dir, 'train')\n",
        "validation_dir = os.path.join(base_dir, 'validation')\n",
        "\n",
        "# Directory with our training cat pictures\n",
        "train_cats_dir = os.path.join(train_dir, 'cats')\n",
        "print ('Total training cat images:', len(os.listdir(train_cats_dir)))\n",
        "\n",
        "# Directory with our training dog pictures\n",
        "train_dogs_dir = os.path.join(train_dir, 'dogs')\n",
        "print ('Total training dog images:', len(os.listdir(train_dogs_dir)))\n",
        "\n",
        "# Directory with our validation cat pictures\n",
        "validation_cats_dir = os.path.join(validation_dir, 'cats')\n",
        "print ('Total validation cat images:', len(os.listdir(validation_cats_dir)))\n",
        "\n",
        "# Directory with our validation dog pictures\n",
        "validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n",
        "print ('Total validation dog images:', len(os.listdir(validation_dogs_dir)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wvidPx6jeFzf"
      },
      "source": [
        "### Create Image Data Generator with Image Augmentation\n",
        "\n",
        "We will use ImageDataGenerator to rescale the images.\n",
        "\n",
        "To create the train generator, specify where the train dataset directory, image size, batch size and binary classification mode.\n",
        "\n",
        "The validation generator is created the same way."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y3PM6GVHcC31"
      },
      "outputs": [],
      "source": [
        "image_size = 160 # All images will be resized to 160x160\n",
        "batch_size = 32\n",
        "\n",
        "# Rescale all images by 1./255 and apply image augmentation\n",
        "train_datagen = keras.preprocessing.image.ImageDataGenerator(\n",
        "                rescale=1./255)\n",
        "\n",
        "validation_datagen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)\n",
        "\n",
        "# Flow training images in batches of 20 using train_datagen generator\n",
        "train_generator = train_datagen.flow_from_directory(\n",
        "                train_dir,  # Source directory for the training images\n",
        "                target_size=(image_size, image_size),\n",
        "                batch_size=batch_size,\n",
        "                # Since we use binary_crossentropy loss, we need binary labels\n",
        "                class_mode='binary')\n",
        "\n",
        "# Flow validation images in batches of 20 using test_datagen generator\n",
        "validation_generator = validation_datagen.flow_from_directory(\n",
        "                validation_dir, # Source directory for the validation images\n",
        "                target_size=(image_size, image_size),\n",
        "                batch_size=batch_size,\n",
        "                class_mode='binary')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OkH-kazQecHB"
      },
      "source": [
        "## Create the base model from the pre-trained convnets\n",
        "We will create the base model from the **MobileNet V2** model developed at Google, and pre-trained on the ImageNet dataset, a large dataset of 1.4M images and 1000 classes of web images. This is a powerful model. Let's see what the features that it has learned can do for our cat vs. dog problem.\n",
        "\n",
        "First, we need to pick which intermediate layer of MobileNet V2 we will use for feature extraction. A common practice is to use the output of the very last layer before the flatten operation, the so-called \"bottleneck layer\". The reasoning here is that the following fully-connected layers will be too specialized to the task the network was trained on, and thus the features learned by these layers won't be very useful for a new task. The bottleneck features, however, retain much generality.\n",
        "\n",
        "Let's instantiate an MobileNet V2 model pre-loaded with weights trained on ImageNet. By specifying the **include_top=False** argument, we load a network that doesn't include the classification layers at the top, which is ideal for feature extraction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "19IQ2gqneqmS"
      },
      "outputs": [],
      "source": [
        "IMG_SHAPE = (image_size, image_size, 3)\n",
        "\n",
        "# Create the base model from the pre-trained model MobileNet V2\n",
        "base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,\n",
        "                                               include_top=False,\n",
        "                                               weights='imagenet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rlx56nQtfe8Y"
      },
      "source": [
        "## Feature extraction\n",
        "We will freeze the convolutional base created from the previous step and use that as a feature extractor, add a classifier on top of it and train the top-level classifier."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CnMLieHBCwil"
      },
      "source": [
        "### Freeze the convolutional base\n",
        "It's important to freeze the convolutional based before we compile and train the model. By freezing (or setting `layer.trainable = False`), we prevent the weights in these layers from being updated during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "OTCJH4bphOeo"
      },
      "outputs": [],
      "source": [
        "base_model.trainable = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "KpbzSmPkDa-N"
      },
      "outputs": [],
      "source": [
        "# Let's take a look at the base model architecture\n",
        "base_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wdMRM8YModbk"
      },
      "source": [
        "#### Add a classification head"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0iqnBeZrfoIc"
      },
      "source": [
        "Now let's add a few layers on top of the base model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "eApvroIyn1K0"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "  base_model,\n",
        "  keras.layers.GlobalAveragePooling2D(),\n",
        "  keras.layers.Dense(1, activation='sigmoid')\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g0ylJXE_kRLi"
      },
      "source": [
        "### Compile the model\n",
        "\n",
        "You must compile the model before training it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RpR8HdyMhukJ"
      },
      "outputs": [],
      "source": [
        "model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.0001),\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "I8ARiyMFsgbH"
      },
      "outputs": [],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lxOcmVr0ydFZ"
      },
      "source": [
        "These 1.2K trainable parameters are divided among 2 TensorFlow `Variable` objects, the weights and biases of the two dense layers:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "krvBumovycVA"
      },
      "outputs": [],
      "source": [
        "len(model.trainable_variables)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RxvgOYTDSWTx"
      },
      "source": [
        "### Train the model\n",
        "\n",
        "After training for 10 epochs, we are able to get ~94% accuracy.\n",
        "\n",
        "If you have more time, train it to convergence (50 epochs, ~96% accuracy)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Om4O3EESkab1"
      },
      "outputs": [],
      "source": [
        "epochs = 10\n",
        "steps_per_epoch = train_generator.n // batch_size\n",
        "validation_steps = validation_generator.n // batch_size\n",
        "\n",
        "history = model.fit_generator(train_generator,\n",
        "                              steps_per_epoch = steps_per_epoch,\n",
        "                              epochs=epochs,\n",
        "                              workers=4,\n",
        "                              validation_data=validation_generator,\n",
        "                              validation_steps=validation_steps)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Hd94CKImf8vi"
      },
      "source": [
        "### Learning curves\n",
        "\n",
        "Let's take a look at the learning curves of the training and validation accuracy / loss, when using the MobileNet V2 base model as a fixed feature extractor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l7HOsQTPNgO9"
      },
      "source": [
        "If you train to convergence (`epochs=50`) the resulting graph should look like this:\n",
        "\n",
        "![Before fine tuning, the model reaches 96% accuracy](images/before_fine_tuning.pnd)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "53OTCh3jnbwV"
      },
      "outputs": [],
      "source": [
        "acc = history.history['acc']\n",
        "val_acc = history.history['val_acc']\n",
        "\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.subplot(2, 1, 1)\n",
        "plt.plot(acc, label='Training Accuracy')\n",
        "plt.plot(val_acc, label='Validation Accuracy')\n",
        "plt.legend(loc='lower right')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.ylim([min(plt.ylim()),1])\n",
        "plt.title('Training and Validation Accuracy')\n",
        "\n",
        "plt.subplot(2, 1, 2)\n",
        "plt.plot(loss, label='Training Loss')\n",
        "plt.plot(val_loss, label='Validation Loss')\n",
        "plt.legend(loc='upper right')\n",
        "plt.ylabel('Cross Entropy')\n",
        "plt.ylim([0,max(plt.ylim())])\n",
        "plt.title('Training and Validation Loss')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CqwV-CRdS6Nv"
      },
      "source": [
        "## Fine tuning\n",
        "In our feature extraction experiment, we were only training a few layers on top of an MobileNet V2 base model. The weights of the pre-trained network were **not** updated during training. One way to increase performance even further is to \"fine-tune\" the weights of the top layers of the pre-trained model alongside the training of the top-level classifier. The training process will force the weights to be tuned from generic features maps to features associated specifically to our dataset.\n",
        "\n",
        "Note: this should only be attempted after you have trained the top-level classifier with the pre-trained model set to non-trainable. If you add a randomly initialized classifier on top of a pre-trained model and attempt to train all layers jointly, the magnitude of the gradient updates will be too large (due to the random weights from the classifier) and your pre-trained model will just forget everything it has learned.\n",
        "\n",
        "Additionally, the reasoning behind fine-tuning the top layers of the pre-trained model rather than all layers of the pre-trained model is the following: in a convnet, the higher up a layer is, the more specialized it is. The first few layers in a convnet learned very simple and generic features, which generalize to almost all types of images. But as you go higher up, the features are increasingly more specific to the dataset that the model was trained on. The goal of fine-tuning is to adapt these specialized features to work with the new dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CPXnzUK0QonF"
      },
      "source": [
        "### Un-freeze the top layers of the model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rfxv_ifotQak"
      },
      "source": [
        "All we need to do is unfreeze the `base_model`, and set the bottom layers be un-trainable. Then, recompile the model (necessary for these changes to take effect), and resume training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "4nzcagVitLQm"
      },
      "outputs": [],
      "source": [
        "base_model.trainable = True"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-4HgVAacRs5v"
      },
      "outputs": [],
      "source": [
        "# Let's take a look to see how many layers are in the base model\n",
        "print(\"Number of layers in the base model: \", len(base_model.layers))\n",
        "\n",
        "# Fine tune from this layer onwards\n",
        "fine_tune_at = 100\n",
        "\n",
        "# Freeze all the layers before the `fine_tune_at` layer\n",
        "for layer in base_model.layers[:fine_tune_at]:\n",
        "  layer.trainable =  False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4Uk1dgsxT0IS"
      },
      "source": [
        "### Compile the model\n",
        "\n",
        "Compile the model using a much-lower training rate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "NtUnaz0WUDva"
      },
      "outputs": [],
      "source": [
        "\n",
        "model.compile(loss='binary_crossentropy',\n",
        "              optimizer = tf.keras.optimizers.RMSprop(lr=2e-5),\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "WwBWy7J2kZvA"
      },
      "outputs": [],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "bNXelbMQtonr"<commit_msg>Reorder Model#compile params in later call to match earlier call for readability<commit_after>{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "77gENRVX40S7"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "d8jyt37T42Vf"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "aPxHdjwW5P2j"
      },
      "outputs": [],
      "source": [
        "#@title MIT License\n",
        "#\n",
        "# Copyright (c) 2017 <NAME>                                                                                                                    # IGNORE_COPYRIGHT: cleared by OSS licensing\n",
        "#\n",
        "# Permission is hereby granted, free of charge, to any person obtaining a\n",
        "# copy of this software and associated documentation files (the \"Software\"),\n",
        "# to deal in the Software without restriction, including without limitation\n",
        "# the rights to use, copy, modify, merge, publish, distribute, sublicense,\n",
        "# and/or sell copies of the Software, and to permit persons to whom the\n",
        "# Software is furnished to do so, subject to the following conditions:\n",
        "#\n",
        "# The above copyright notice and this permission notice shall be included in\n",
        "# all copies or substantial portions of the Software.\n",
        "#\n",
        "# THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
        "# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
        "# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL\n",
        "# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
        "# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING\n",
        "# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER\n",
        "# DEALINGS IN THE SOFTWARE."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hRTa3Ee15WsJ"
      },
      "source": [
        "# Transfer Learning Using Pretrained ConvNets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dQHMcypT3vDT"
      },
      "source": [
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/tutorials/images/transfer_learning\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003eView on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/images/transfer_learning.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/tutorials/images/transfer_learning.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2X4KyhORdSeO"
      },
      "source": [
        "In this tutorial we will discuss how to classify cats vs dogs images by using transfer learning from a pre-trained network. This will allows us to get higher accuracies than we saw by training our network from scratch.\n",
        "\n",
        "A **pre-trained model **is a saved network that was previously trained on a large dataset, typically on a large-scale image-classification task. We can either use the pretrained model as it is or transfer learning using the pretrained convents. The intuition behind **transfer learning **is that if this model trained on a large and general enough dataset, this model will effectively serve as a generic model of the visual world. We can leverage these learned feature maps without having to train a large model on a large dataset by using these models as the basis of our own model specific to our task. There are 2 scenarios of transfer learning using a pretrained model:\n",
        "\n",
        "1. **Feature Extraction** - use the representations of learned by a previous network to extract meaningful features from new samples. We simply add a new classifier, which will be trained from scratch, on top of the pretrained model so that we can repurpose the feature maps learned previously for our dataset. **Do we use the entire pretrained model or just the convolutional base?** - We use the feature extraction portion of these pretrained convnets (convolutional base) since they are likely to be generic features and learned concepts over a picture. However, the classification part of the pretrained model is often specific to original classification task, and subsequently specific to the set of classes on which the model was trained.\n",
        "2. **Fine-Tuning** - unfreezing a few of the top layers of a frozen model base used for feature extraction, and jointly training both the newly added classifier layers as well as the last layers of the frozen model. This allows us to \"fine tune\" the higher order feature representations in addition to our final classifier in order to make them more relevant for the specific task involved.\n",
        "\n",
        "**We will follow the general machine learning workflow:**\n",
        "1. Examine and understand data\n",
        "2. Build an input pipeline - using Keras ImageDataGenerator as we did in the image classification tutorial\n",
        "3. Compose our model\n",
        "  * Load in our pretrained model (and pretrained weights)\n",
        "  * Stack our classification layers on top\n",
        "4. Train our model\n",
        "5. Evaluate model\n",
        "\n",
        "We will see an example of using the pre-trained convnet as the feature extraction and then fine-tune to train the last few layers of the base model.\n",
        "\n",
        "**Audience:** This post is geared towards beginners with some Keras API and ML background. To get the most out of this post, you should have some basic ML background, know what CNNs are, and be familiar with the Keras Sequential API.\n",
        "\n",
        "**Time Estimated**: 30 minutes"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "iBMcobPHdD8O"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "\n",
        "import os\n",
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "print(\"TensorFlow version is \", tf.__version__)\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib.image as mpimg"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "v77rlkCKW0IJ"
      },
      "source": [
        "## Data preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "aXzwKdouXf1h"
      },
      "source": [
        "### Download data - cats_and_dogs_filtered.zip\n",
        "We will download a filtered version of Kaggle's [Dogs vs Cats](https://www.kaggle.com/c/dogs-vs-cats/data) dataset. Then store the downloaded zip file to the \"/tmp/\" directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "nRnO59Kr6enO"
      },
      "outputs": [],
      "source": [
        "zip_file = tf.keras.utils.get_file(origin=\"https://storage.googleapis.com/mledu-datasets/cats_and_dogs_filtered.zip\",\n",
        "                                   fname=\"cats_and_dogs_filtered.zip\", extract=True)\n",
        "base_dir, _ = os.path.splitext(zip_file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9_6h-c5EXN91"
      },
      "source": [
        "### Prepare training and validation cats and dogs datasets\n",
        "Create the training and validation directories for cats datasets and dog datasets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RWcldM4TXLen"
      },
      "outputs": [],
      "source": [
        "train_dir = os.path.join(base_dir, 'train')\n",
        "validation_dir = os.path.join(base_dir, 'validation')\n",
        "\n",
        "# Directory with our training cat pictures\n",
        "train_cats_dir = os.path.join(train_dir, 'cats')\n",
        "print ('Total training cat images:', len(os.listdir(train_cats_dir)))\n",
        "\n",
        "# Directory with our training dog pictures\n",
        "train_dogs_dir = os.path.join(train_dir, 'dogs')\n",
        "print ('Total training dog images:', len(os.listdir(train_dogs_dir)))\n",
        "\n",
        "# Directory with our validation cat pictures\n",
        "validation_cats_dir = os.path.join(validation_dir, 'cats')\n",
        "print ('Total validation cat images:', len(os.listdir(validation_cats_dir)))\n",
        "\n",
        "# Directory with our validation dog pictures\n",
        "validation_dogs_dir = os.path.join(validation_dir, 'dogs')\n",
        "print ('Total validation dog images:', len(os.listdir(validation_dogs_dir)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wvidPx6jeFzf"
      },
      "source": [
        "### Create Image Data Generator with Image Augmentation\n",
        "\n",
        "We will use ImageDataGenerator to rescale the images.\n",
        "\n",
        "To create the train generator, specify where the train dataset directory, image size, batch size and binary classification mode.\n",
        "\n",
        "The validation generator is created the same way."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y3PM6GVHcC31"
      },
      "outputs": [],
      "source": [
        "image_size = 160 # All images will be resized to 160x160\n",
        "batch_size = 32\n",
        "\n",
        "# Rescale all images by 1./255 and apply image augmentation\n",
        "train_datagen = keras.preprocessing.image.ImageDataGenerator(\n",
        "                rescale=1./255)\n",
        "\n",
        "validation_datagen = keras.preprocessing.image.ImageDataGenerator(rescale=1./255)\n",
        "\n",
        "# Flow training images in batches of 20 using train_datagen generator\n",
        "train_generator = train_datagen.flow_from_directory(\n",
        "                train_dir,  # Source directory for the training images\n",
        "                target_size=(image_size, image_size),\n",
        "                batch_size=batch_size,\n",
        "                # Since we use binary_crossentropy loss, we need binary labels\n",
        "                class_mode='binary')\n",
        "\n",
        "# Flow validation images in batches of 20 using test_datagen generator\n",
        "validation_generator = validation_datagen.flow_from_directory(\n",
        "                validation_dir, # Source directory for the validation images\n",
        "                target_size=(image_size, image_size),\n",
        "                batch_size=batch_size,\n",
        "                class_mode='binary')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "OkH-kazQecHB"
      },
      "source": [
        "## Create the base model from the pre-trained convnets\n",
        "We will create the base model from the **MobileNet V2** model developed at Google, and pre-trained on the ImageNet dataset, a large dataset of 1.4M images and 1000 classes of web images. This is a powerful model. Let's see what the features that it has learned can do for our cat vs. dog problem.\n",
        "\n",
        "First, we need to pick which intermediate layer of MobileNet V2 we will use for feature extraction. A common practice is to use the output of the very last layer before the flatten operation, the so-called \"bottleneck layer\". The reasoning here is that the following fully-connected layers will be too specialized to the task the network was trained on, and thus the features learned by these layers won't be very useful for a new task. The bottleneck features, however, retain much generality.\n",
        "\n",
        "Let's instantiate an MobileNet V2 model pre-loaded with weights trained on ImageNet. By specifying the **include_top=False** argument, we load a network that doesn't include the classification layers at the top, which is ideal for feature extraction."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "19IQ2gqneqmS"
      },
      "outputs": [],
      "source": [
        "IMG_SHAPE = (image_size, image_size, 3)\n",
        "\n",
        "# Create the base model from the pre-trained model MobileNet V2\n",
        "base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,\n",
        "                                               include_top=False,\n",
        "                                               weights='imagenet')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rlx56nQtfe8Y"
      },
      "source": [
        "## Feature extraction\n",
        "We will freeze the convolutional base created from the previous step and use that as a feature extractor, add a classifier on top of it and train the top-level classifier."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CnMLieHBCwil"
      },
      "source": [
        "### Freeze the convolutional base\n",
        "It's important to freeze the convolutional based before we compile and train the model. By freezing (or setting `layer.trainable = False`), we prevent the weights in these layers from being updated during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "OTCJH4bphOeo"
      },
      "outputs": [],
      "source": [
        "base_model.trainable = False"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "KpbzSmPkDa-N"
      },
      "outputs": [],
      "source": [
        "# Let's take a look at the base model architecture\n",
        "base_model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wdMRM8YModbk"
      },
      "source": [
        "#### Add a classification head"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0iqnBeZrfoIc"
      },
      "source": [
        "Now let's add a few layers on top of the base model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "eApvroIyn1K0"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "  base_model,\n",
        "  keras.layers.GlobalAveragePooling2D(),\n",
        "  keras.layers.Dense(1, activation='sigmoid')\n",
        "])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g0ylJXE_kRLi"
      },
      "source": [
        "### Compile the model\n",
        "\n",
        "You must compile the model before training it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "RpR8HdyMhukJ"
      },
      "outputs": [],
      "source": [
        "model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=0.0001),\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "I8ARiyMFsgbH"
      },
      "outputs": [],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lxOcmVr0ydFZ"
      },
      "source": [
        "These 1.2K trainable parameters are divided among 2 TensorFlow `Variable` objects, the weights and biases of the two dense layers:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "krvBumovycVA"
      },
      "outputs": [],
      "source": [
        "len(model.trainable_variables)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "RxvgOYTDSWTx"
      },
      "source": [
        "### Train the model\n",
        "\n",
        "After training for 10 epochs, we are able to get ~94% accuracy.\n",
        "\n",
        "If you have more time, train it to convergence (50 epochs, ~96% accuracy)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Om4O3EESkab1"
      },
      "outputs": [],
      "source": [
        "epochs = 10\n",
        "steps_per_epoch = train_generator.n // batch_size\n",
        "validation_steps = validation_generator.n // batch_size\n",
        "\n",
        "history = model.fit_generator(train_generator,\n",
        "                              steps_per_epoch = steps_per_epoch,\n",
        "                              epochs=epochs,\n",
        "                              workers=4,\n",
        "                              validation_data=validation_generator,\n",
        "                              validation_steps=validation_steps)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Hd94CKImf8vi"
      },
      "source": [
        "### Learning curves\n",
        "\n",
        "Let's take a look at the learning curves of the training and validation accuracy / loss, when using the MobileNet V2 base model as a fixed feature extractor."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "l7HOsQTPNgO9"
      },
      "source": [
        "If you train to convergence (`epochs=50`) the resulting graph should look like this:\n",
        "\n",
        "![Before fine tuning, the model reaches 96% accuracy](images/before_fine_tuning.pnd)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "53OTCh3jnbwV"
      },
      "outputs": [],
      "source": [
        "acc = history.history['acc']\n",
        "val_acc = history.history['val_acc']\n",
        "\n",
        "loss = history.history['loss']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "plt.figure(figsize=(8, 8))\n",
        "plt.subplot(2, 1, 1)\n",
        "plt.plot(acc, label='Training Accuracy')\n",
        "plt.plot(val_acc, label='Validation Accuracy')\n",
        "plt.legend(loc='lower right')\n",
        "plt.ylabel('Accuracy')\n",
        "plt.ylim([min(plt.ylim()),1])\n",
        "plt.title('Training and Validation Accuracy')\n",
        "\n",
        "plt.subplot(2, 1, 2)\n",
        "plt.plot(loss, label='Training Loss')\n",
        "plt.plot(val_loss, label='Validation Loss')\n",
        "plt.legend(loc='upper right')\n",
        "plt.ylabel('Cross Entropy')\n",
        "plt.ylim([0,max(plt.ylim())])\n",
        "plt.title('Training and Validation Loss')\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CqwV-CRdS6Nv"
      },
      "source": [
        "## Fine tuning\n",
        "In our feature extraction experiment, we were only training a few layers on top of an MobileNet V2 base model. The weights of the pre-trained network were **not** updated during training. One way to increase performance even further is to \"fine-tune\" the weights of the top layers of the pre-trained model alongside the training of the top-level classifier. The training process will force the weights to be tuned from generic features maps to features associated specifically to our dataset.\n",
        "\n",
        "Note: this should only be attempted after you have trained the top-level classifier with the pre-trained model set to non-trainable. If you add a randomly initialized classifier on top of a pre-trained model and attempt to train all layers jointly, the magnitude of the gradient updates will be too large (due to the random weights from the classifier) and your pre-trained model will just forget everything it has learned.\n",
        "\n",
        "Additionally, the reasoning behind fine-tuning the top layers of the pre-trained model rather than all layers of the pre-trained model is the following: in a convnet, the higher up a layer is, the more specialized it is. The first few layers in a convnet learned very simple and generic features, which generalize to almost all types of images. But as you go higher up, the features are increasingly more specific to the dataset that the model was trained on. The goal of fine-tuning is to adapt these specialized features to work with the new dataset."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "CPXnzUK0QonF"
      },
      "source": [
        "### Un-freeze the top layers of the model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rfxv_ifotQak"
      },
      "source": [
        "All we need to do is unfreeze the `base_model`, and set the bottom layers be un-trainable. Then, recompile the model (necessary for these changes to take effect), and resume training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "4nzcagVitLQm"
      },
      "outputs": [],
      "source": [
        "base_model.trainable = True"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-4HgVAacRs5v"
      },
      "outputs": [],
      "source": [
        "# Let's take a look to see how many layers are in the base model\n",
        "print(\"Number of layers in the base model: \", len(base_model.layers))\n",
        "\n",
        "# Fine tune from this layer onwards\n",
        "fine_tune_at = 100\n",
        "\n",
        "# Freeze all the layers before the `fine_tune_at` layer\n",
        "for layer in base_model.layers[:fine_tune_at]:\n",
        "  layer.trainable =  False"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "4Uk1dgsxT0IS"
      },
      "source": [
        "### Compile the model\n",
        "\n",
        "Compile the model using a much-lower training rate."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "NtUnaz0WUDva"
      },
      "outputs": [],
      "source": [
        "\n",
        "model.compile(optimizer = tf.keras.optimizers.RMSprop(lr=2e-5),\n",
        "              loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "WwBWy7J2kZvA"
      },
      "outputs": [],
      "source": [
        "model.summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "bNXelbMQtonr"
{'fraction_non_alphanumeric': 0.13643910910930016, 'fraction_numerical': 0.011641887894345096, 'mean_word_length': 2.0760831278619234, 'pattern_counts': {'":': 542, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 18, 'lorem ipsum': 0, 'www.': 12, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 38, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2644256', 'n_tokens_mistral': 16478, 'n_tokens_neox': 15743, 'n_words': 4640}