<jupyter_start><jupyter_text>T81-558: Applications of Deep Neural Networks**Module 6: Convolutional Neural Networks (CNN) for Computer Vision*** Instructor: [<NAME>](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/).  Module 6 Material* Part 6.1: Image Processing in Python [[Video]](https://www.youtube.com/watch?v=4Bh3gqHkIgc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_1_python_images.ipynb)* Part 6.2: Keras Neural Networks for Digits and Fashion MNIST [[Video]](https://www.youtube.com/watch?v=-SA8BmGvWYE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_2_cnn.ipynb)* Part 6.3: Implementing a ResNet in Keras [[Video]](https://www.youtube.com/watch?v=qMFKsMeE6fM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_3_resnet.ipynb)* Part 6.4: Using Your Own Images with Keras [[Video]](https://www.youtube.com/watch?v=VcFja1fUNSk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_4_keras_images.ipynb)* **Part 6.5: Recognizing Multiple Images with YOLO Darknet** [[Video]](https://www.youtube.com/watch?v=oQcAKvBFli8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_5_yolo.ipynb)  Google CoLab InstructionsThe following code ensures that Google CoLab is running the correct version of TensorFlow.  Running the following code will map your GDrive to ```/content/drive```.<jupyter_code>try:
    from google.colab import drive
    drive.mount('/content/drive', force_remount=True)
    COLAB = True
    print("Note: using Google CoLab")
    %tensorflow_version 2.x
except:
    print("Note: not using Google CoLab")
    COLAB = False

# Nicely formatted time string
def hms_string(sec_elapsed):
    h = int(sec_elapsed / (60 * 60))
    m = int((sec_elapsed % (60 * 60)) / 60)
    s = sec_elapsed % 60
    return f"{h}:{m:>02}:{s:>05.2f}"<jupyter_output><empty_output><jupyter_text>Part 6.5: Recognizing Multiple Images with DarknetProgrammers typically design convolutional neural networks to classify a single item centered in an image.  However, as humans, we can recognize many items in our field of view in real-time.  It is advantageous to be able to recognize multiple items in a single image.  One of the most advanced means of doing this is YOLO DarkNet (not to be confused with the Internet [Darknet](https://en.wikipedia.org/wiki/Darknet).  YOLO [[Cite:redmon2016you]](https://arxiv.org/abs/1506.02640) is an acronym for You Only Look Once.  The fact that YOLO must only look once speaks to the efficiency of the algorithm.  In this context, to "look" means to perform one scan over the image. Figure 6.YOLO shows YOLO tagging in action.**Figure 6.YOLO: YOLO Tagging**![DarkNet](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/darknet-predictions.jpg "DarkNet")It is also possible to run YOLO on live video streams.  Figure 6.VIDEO is from the YouTube Video for this module.**Figure 6.VIDEO: YOLO Video Tagging**![DarkNet Video](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/yolo_video.png "DarkNet Video")As you can see, it is classifying many things in this video.  My collection of books behind me is adding considerable "noise," as DarkNet tries to classify every book behind me.  If you watch the video, you can see that it is less than perfect.  The coffee mug that I pick up gets classified as a cell phone and, at times, a remote.  The small yellow object behind me on the desk is a small toolbox (not a remote).  However, it gets classified as a book at times and a remote at other times.  Currently, this algorithm classifies each frame on its own.  The program could achieve greater accuracy if it analyzed multiple images from a video stream. Consider when you see an object coming towards you, if it changes angles, you might form a better opinion of what it was.  If that same object now changes to an unfavorable angle, you still know what it is, based on previous information. How Does DarkNet/YOLO Work?YOLO begins by resizing the image to a $S \times S$ grid.  YOLO runs a  single convolutional neural network against this grid that predicts bounding boxes and what might be contained by those boxes.  Each bounding box also has a confidence in which item it believes the box contains.  YOLO is a regular convolution network, just like we've seen previously.  The only difference is that a YOLO CNN outputs multiple prediction bounding boxes. At a high level, Figure 6.YOLO-DET illustrates this.**Figure 6.YOLO-DET: The YOLO Detection System**![The YOLO Detection System](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/yolo-1.png "The YOLO Detection System")The output of the YOLO convolutional neural networks is essentially a multiple regression.  YOLO generated the following values for each of the bounding rectangles.* **x** - The x-coordinate of the center of a bounding rectangle.* **y** - The y-coordinate of the center of a bounding rectangle.* **w** - The width of each bounding rectangle.* **h** - The height of each bounding rectangle.* **labels** - The relative probabilities of each of the labels (1 value for each label)* **confidence** - The confidence in this rectangle.The output layer of a Keras neural network is a Tensor.  In the case of YOLO, this output tensor is 3D and is of the following dimensions.$ S \times S \times (B \cdot 5 + C) $The constants in the above expression are:* *S* - The dimensions that YOLO overlays across the source image.* *B* - The number of potential bounding rectangles generated for each grid cell.* *C* - The number of class labels that here are.The value 5 in the above expression is simply the count of non-label components of each bounding rectangle ($x$, $y$, $h$, $w$, $confidence$.Because there are $S^2 \cdot B$ total potential bounding rectangles, the image is nearly full.  Because of this, it is essential to drop all rectangles below some threshold of confidence.  The image below demonstrates this.The actual structure of the convolutional neural network behind YOLO is relatively simple, as the following figure illustrates.  Because there is only one convolutional neural network, and it "only looks once," the performance is not impacted by how many objects are detected.  Figure 6.YOLO-STRUCT shows the YOLO structure.**Figure 6.YOLO-STRUCT: YOLO Structure**![YOLO Structure](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/yolo-3.png "YOLO Structure")Figure 6.YOLO-DEMO shows some additional recognitions performed by a YOLO.**Figure 6.YOLO-DEMO: YOLO Structure**![YOLO Structure](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/yolo-4.png "YOLO Structure") Using YOLO in PythonTo make use of YOLO in Python, you have several options:* **[DarkNet](https://pjreddie.com/darknet/yolo/)** - The original implementation of YOLO, written in C.* **[yolov3-tf2](https://github.com/zzh8829/yolov3-tf2)** - An unofficial Python package that implements YOLO in Python, using TensorFlow 2.0.The code provided in this notebook works equally well when run either locally or from Google CoLab.  In either case, the programmer should use TensorFlow 2.0.  Installing YoloV3-TF2YoloV3-TF2 is not available directly through either PIP or CONDA.  Additionally, YoloV3-TF2 is not installed in Google CoLab by default. Therefore, whether you wish to use YoloV3-TF2 through CoLab or run it locally, you need to go through several steps to install it.  This section describes the process of installing YoloV3-TF2.  The same steps apply to either CoLab or a local install.  For CoLab, you must repeat these steps each time the system restarts your virtual environment.  For a local install, you must perform these steps only once for your virtual Python environment.  If you are installing locally, make sure to install to the same virtual environment that you created for this course.  The following command installs YoloV3-TF2 directly from it's GitHub repository.<jupyter_code>import sys

!{sys.executable} -m pip install git+https://github.com/
        zzh8829/yolov3-tf2.git@master<jupyter_output>Collecting git+https://github.com/zzh8829/yolov3-tf2.git@master
  Cloning https://github.com/zzh8829/yolov3-tf2.git (to revision master) to /private/var/folders/nz/jw1_lq4s10389lbx_j_vjyg4qy7h5_/T/pip-req-build-plnt65lr
  Running command git clone -q https://github.com/zzh8829/yolov3-tf2.git /private/var/folders/nz/jw1_lq4s10389lbx_j_vjyg4qy7h5_/T/pip-req-build-plnt65lr
Requirement already satisfied (use --upgrade to upgrade): yolov3-tf2==0.1 from git+https://github.com/zzh8829/yolov3-tf2.git@master in /Users/jheaton/miniconda3/envs/tensorflow/lib/python3.7/site-packages
Building wheels for collected packages: yolov3-tf2
  Building wheel for yolov3-tf2 (setup.py) ... [?25ldone
[?25h  Created wheel for yolov3-tf2: filename=yolov3_tf2-0.1-cp37-none-any.whl size=9193 sha256=f8aa39a7acb8d7a766475aa5779bb7ae5831ef15eff8442eace8ccf04f97f74c
  Stored in directory: /private/var/folders/nz/jw1_lq4s10389lbx_j_vjyg4qy7h5_/T/pip-ephem-wheel-cache-6ks2ceq_/wheels/59/1b/97/905ab51e9c0330efe8c3c518[...]<jupyter_text>Before you can make use of YoloV3-TF2 there are several files you must obtain:* **yolov3.weights** - These are the pre-trained weights provided by the author of YOLO.* **convert.py** - This is a Python script that converts **yolov3.weights** into a TensorFlow compatible weight format.* **coco.names** - The names of the 80 items that the **yolov3.weights** neural network was trained to recognize.* **yolov3.tf** - The YOLO weights converted to a format that TensorFlow can use directly.The code provided below obtains these files.  The script stores these files to either your GDrive, if you are using CoLab, or a local folder named "data" if you are running locally.Researchers have trained YOLO on a variety of different computer image datasets.  The version of YOLO weights used in this course is from the dataset Common Objects in Context (COCO). [[Cite: lin2014microsoft]](https://arxiv.org/abs/1405.0312) This dataset contains images labeled into 80 different classes.  COCO is the source of the file coco.txt that used in this module.  Developers have also adapted YOLO for mobile devices by creating the YOLO Tiny pre-trained weights that use a much smaller convolutional neural network and still achieve acceptable levels of quality.  Though YoloV3-TF2 can work with either YOLO Tiny or regular YOLO we are not using the tiny weights for this course.<jupyter_code>import tensorflow as tf
import os

if COLAB:
  ROOT = '/content/drive/My Drive/projects/t81_558_dlearning/yolo'
else:
  ROOT = os.path.join(os.getcwd(),'data')

filename_darknet_weights = tf.keras.utils.get_file(
    os.path.join(ROOT,'yolov3.weights'),
    origin='https://pjreddie.com/media/files/yolov3.weights')
TINY = False

filename_convert_script = tf.keras.utils.get_file(
    os.path.join(os.getcwd(),'convert.py'),
    origin='https://raw.githubusercontent.com/zzh8829/'\
        "yolov3-tf2/master/convert.py')

filename_classes = tf.keras.utils.get_file(
    os.path.join(ROOT,'coco.names'),
    origin='https://raw.githubusercontent.com/zzh8829/'\
       'yolov3-tf2/master/data/coco.names')
filename_converted_weights = os.path.join(ROOT,'yolov3.tf')<jupyter_output>Downloading data from https://raw.githubusercontent.com/zzh8829/yolov3-tf2/master/convert.py
8192/1067 [======================================================================================================================================================================================================================================] - 0s 0us/step<jupyter_text>Transfering WeightsIn the course, we transfer already trained weights into our YOLO networks.  It can take considerable time to train a YOLO network from scratch.  If you would like to train a YOLO network to recognize images other than the COLO provided images, then you may need to train your own YOLO information.  If training from scratch is something you need to do, there is further information on this at the YoloV3-TF2 GitHub repository.The weights provided by the original authors of YOLO is not directly compatible with TensorFlow.  Because of this, it is necessary first to convert the YOLO provided weights into a TensorFlow compatible format. The following code does this conversion.  This process does not need to be repeated by the program. Once the conversion script processes YOLO weights the saved to the yolov3.tf YOLO can reuse these converted wights. The following code performs this conversion.<jupyter_code>import sys
!{sys.executable} "{filename_convert_script}" --weights 
    "{filename_darknet_weights}" --output "{filename_converted_weights}"<jupyter_output>2019-12-24 20:08:13.233259: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2019-12-24 20:08:13.233578: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 8. Tune using inter_op_parallelism_threads for best performance.
Model: "yolov3"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_1 (InputLayer)            [(None, None, None,  0                                            
______________________________________________[...]<jupyter_text>The conversion script is no longer needed once this script converts the YOLO weights have to a TensorFlow format.  Because this executable file resides in the same directory as the course files, we delete it at this point.<jupyter_code>import os
os.remove(filename_convert_script)<jupyter_output><empty_output><jupyter_text>Now that we have all of the files needed for YOLO, we are ready to use it to recognize components of an image.  Running DarkFlow (YOLO)The YoloV3-TF2 library can easily integrate with Python applications.  The initialization of the library consists of three steps.  First, it is essential to import all of the needed packages for the library.  Next, the Python program must define all of the YOLO configurations through the Keras flags architecture. The Keras flag system primarily works from the command line; however, it also allows configuration programmatically in an application.  For this example, we configure the package programmatically.  Finally, we must scan available devices so that our application takes advantage of any GPUs.   The following code performs all three of these steps.<jupyter_code>import time
from absl import app, flags, logging
from absl.flags import FLAGS
import cv2
import numpy as np
import tensorflow as tf
from yolov3_tf2.models import (YoloV3, YoloV3Tiny)
from yolov3_tf2.dataset import transform_images, load_tfrecord_dataset
from yolov3_tf2.utils import draw_outputs
import sys
from PIL import Image, ImageFile
import requests

# Flags are used to define several options for YOLO.
flags.DEFINE_string('classes', filename_classes, 'path to classes file')
flags.DEFINE_string('weights', filename_converted_weights, 'path to weights file')
flags.DEFINE_boolean('tiny', False, 'yolov3 or yolov3-tiny')
flags.DEFINE_integer('size', 416, 'resize images to')
flags.DEFINE_string('tfrecord', None, 'tfrecord instead of image')
flags.DEFINE_integer('num_classes', 80, 'number of classes in the model')
FLAGS([sys.argv[0]])

# Locate devices to run YOLO on (e.g. GPU)
physical_devices = tf.config.experimental.list_physical_devices('GPU')
if len(physical_devices) > 0:
    tf.config.experimental.set_memory_growth(physical_devices[0], True)<jupyter_output><empty_output><jupyter_text>It is important to understand that Keras flags can only be defined once. If you are going to classify more than one image, make sure that you do not define the flags additional times.The following code initializes a YoloV3-TF2 classification object.  The weights are loaded, and the object is ready for use as the **yolo** variable.  It is not necessary to reload the weights and obtain a new **yolo** variable for each classification.<jupyter_code># This example does not use the "Tiny version"
if FLAGS.tiny:
    yolo = YoloV3Tiny(classes=FLAGS.num_classes)
else:
    yolo = YoloV3(classes=FLAGS.num_classes)

# Load weights and classes
yolo.load_weights(FLAGS.weights).expect_partial()
print('weights loaded')

class_names = [c.strip() for c in open(FLAGS.classes).readlines()]
print('classes loaded')<jupyter_output>weights loaded
classes loaded<jupyter_text>Next, we obtain an image to classify. For this example, the program loads the image from a URL.  YoloV3-TF2 expects that the image is in the format of a Numpy array.  An image file, such as JPEG or PNG, is converted into this raw Numpy format by calling the TensorFlow **decode_image** function.  YoloV3-TF2 can obtain images from other sources, so long as the program first decodes them to raw Numpy format.  The following code obtains the image in this format.<jupyter_code># Read image to classify
url = "https://raw.githubusercontent.com/jeffheaton/"\
    "t81_558_deep_learning/master/images/cook.jpg"
response = requests.get(url)
img_raw = tf.image.decode_image(response.content, channels=3)<jupyter_output><empty_output><jupyter_text>At this point, we can classify the image that was just loaded.  The program should preprocess the image so that it is the size expected by YoloV3-TF2.  Your program also sets the confidence threshold at this point.  Any sub-image recognized with confidence below this value is not returned by YOLO.<jupyter_code># Preprocess image
img = tf.expand_dims(img_raw, 0)
img = transform_images(img, FLAGS.size)

# Desired threshold (any sub-image below this confidence 
# level will be ignored.)
FLAGS.yolo_score_threshold = 0.5

# Recognize and report results
t1 = time.time()
boxes, scores, classes, nums = yolo(img)
t2 = time.time()
print(f"Prediction time: {hms_string(t2 - t1)}")<jupyter_output>Prediction time: 0:00:02.18<jupyter_text>It is important to note that the **yolo** class instantiated here is a callable object, which means that it can fill the role of both an object and a function. Acting as a function, *yolo* returns three arrays named **boxes**, **scores**, and **classes** that are of the same length.  The function returns all sub-images found with a score above the minimum threshold.  Additionally, the **yolo** function returns an array named called **nums**. The first element of the **nums** array specifies how many sub-images YOLO found to be above the score threshold.* **boxes** - The bounding boxes for each of the sub-images detected in the image sent to YOLO.* **scores** - The confidence for each of the sub-images detected.* **classes** - The string class names for each of the items.  These are COCO names such as "person" or "dog." * **nums** - The number of images above the threshold.Your program should use these values to perform whatever actions you wish as a result of the input image.  The following code simply displays the images detected above the threshold.<jupyter_code>print('detections:')
for i in range(nums[0]):
    cls = class_names[int(classes[0][i])]
    score = np.array(scores[0][i])
    box = np.array(boxes[0][i])
    print(f"\t{cls}, {score}, {box}")<jupyter_output>detections:
	person, 0.9995919466018677, [0.31659657 0.10725152 0.68426734 0.74258995]
	dog, 0.9896982312202454, [0.51111007 0.55769485 0.933974   0.8187981 ]
	microwave, 0.9839581847190857, [0.0069517  0.08101548 0.27909756 0.28820014]
	oven, 0.9383126497268677, [0.00773552 0.32521245 0.42321444 0.83368266]
	bottle, 0.8538918495178223, [0.73093545 0.23399045 0.76463544 0.32874537]
	bottle, 0.5538208484649658, [0.790116   0.26327905 0.8189085  0.3274593 ]<jupyter_text>Your program should use these values to perform whatever actions you wish as a result of the input image.  The following code simply displays the images detected above the threshold.YoloV3-TF2 includes a function named **draw_outputs** that allows the sub-image detections to visualized.  The following image shows the output of the draw_outputs function.  You might have first seen YOLO demonstrated as an image with boxes and labels around the sub-images. A program can produce this output with the arrays returned by the **yolo** function.<jupyter_code># Display image using YOLO library's built in function
img = img_raw.numpy()
img = draw_outputs(img, (boxes, scores, classes, nums), class_names)
#cv2.imwrite(FLAGS.output, img) # Save the image
display(Image.fromarray(img, 'RGB')) # Display the image<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09626552098578915, 'fraction_numerical': 0.039138850856900054, 'mean_word_length': 5.031321184510251, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 43, 'https://': 28, 'lorem ipsum': 0, 'www.': 5, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '39255', 'license': 'Apache-2.0', 'n_tokens_mistral': 6737, 'n_tokens_neox': 6203, 'n_words': 2420, 'path': 't81_558_class_06_5_yolo.ipynb', 'repo_name': 'englyndaomanu/t81_558_deep_learning', 'size': '21186'}