<jupyter_start><jupyter_text>Gap Framework - Computer VisionIn this session, we will introduce you to preprocessing image data for computer vision. Preprocessing, storage, retrieval and batch management are all handled by two classes, the Image and Images class.    Image - represents a single preprocessed image    Images - represents a collection (or batch) of preprocessed images<jupyter_code># Let's go the directory of the Gap Framework
import os
os.chdir("../")
%ls<jupyter_output><empty_output><jupyter_text>SetupLet's start by importing the Gap vision module.<jupyter_code># import the Gap Vision module
from gapml.vision import Image, Images<jupyter_output><empty_output><jupyter_text>Let's go to a respository of images for sign language. We will use this repository for image preprocessing for computer vision.<jupyter_code>from urllib import request
import zipfile

# NOTE: This dataset can be downloaded from www.labs.earth/datasets/sign-lang.zip
os.makedirs("../Training/AITraining/Intermediate/Machine Learning/sign-lang", exist_ok=True)
os.chdir("../Training/AITraining/Intermediate/Machine Learning/sign-lang")

if not os.path.isdir('gestures'):
    url = 'http://www.labs.earth/datasets/sign-lang.zip'
    request.urlretrieve(url, 'sign-lang.zip')
    #unzip file
    with zipfile.ZipFile('sign-lang.zip', 'r') as zip_ref:
        zip_ref.extractall('./')
    # delete file
    os.remove('sign-lang.zip')
    print('done!')
# The sign language characters (a-z) are labeled 1 .. 26, and 0 is for not a character.
# Each of the training images are under a subdirectory of the corresponding label.
labels = os.listdir("gestures")
print(labels)<jupyter_output><empty_output><jupyter_text>Image ClassThe Image class supports the preprocessing of a single image into machine learning ready data. It can process JPG, PNG, TIF and GIF files. We will start by instantiating an Image object for an image in the sign-lang image collection. For parameters, we will give the path to the image, and the label value (1). Labels must be mapped into integer values. For the sign-lang dataset, 1-26 is mapped to the 26 letters of the alphabet, and 0 is for a non-letter. When we instantiate the image, by default the following will happen:    1. The image is read in and decompressed, as a numpy array.    2. The image is processed according to the configuration parameters or defaults (e.g., resize, normalized, flatten,          channel conversion).    3. The raw image data, processed image data, thumbnail, and metadata are stored to a HDF5 file system.<jupyter_code>image = Image('gestures/1/1.jpg', 1)<jupyter_output><empty_output><jupyter_text>Image PropertiesLet's look at some properties of the Image class.Note how the shape of the ML ready data is (50, 50, 3). We will change that in a bit.<jupyter_code>print( image.name )   # The root name of the image (w/o suffix)
print( image.type )   # Type of image (e.g. jpeg)
print( image.dir )    # The directory where the ML ready data will be stored
print( image.size )   # The original size of the image
print( image.shape )  # The shape of the preprocessed image (ML ready data)
print( image.label )  # The label
print( image.time )   # The amount of time (secs) to preprocess the image<jupyter_output><empty_output><jupyter_text>Image DataLet's look at both the raw and ML ready data.<jupyter_code>print("Raw Data", image.raw)
print("ML ready data", image.data)<jupyter_output><empty_output><jupyter_text>By default, the number of channels is preserved (e.g., 1 for grayscale, 3 for RGB, and 4 for RGBA), and the data is normalized. On the later, the 0..255 pixel values are rescaled between 0 and 1.Let's now change the preprocessing of the image to a grayscale image and resize it to 32x32. When we print the shape, you can see the 3rd dimension (channels) is gone - indicating a grayscale image, and the size is now 32 by 32.<jupyter_code>image = Image('gestures/1/1.jpg', 1, config=['grayscale', 'resize=(32,32)'])

print( image.shape )<jupyter_output><empty_output><jupyter_text>Let's now say that the image data will be feed into a ANN (not CNN) or to a CNN with a 1D input vector. In this case, we need to feed the ML ready data as a flatten 1D vector. We can do that to. Now when we print the shape you can see its 1024 (32 x 32).<jupyter_code>image = Image('gestures/1/1.jpg', 1, config=['grayscale', 'resize=(32,32)', 'flatten', 'raw'])

print( image.shape )<jupyter_output><empty_output><jupyter_text>Image LoadingWhen an image is preprocessed, the ML ready data, raw data and attributes are stored in an HDF5 file. We can subsequently recall (load) the image information from the HDF5 file into an Image object.<jupyter_code>image = Image()   # Create an empty image object
image.load('1.h5')<jupyter_output><empty_output><jupyter_text>Let's see if we get the same properties again.<jupyter_code>print( image.name )   # The root name of the image (w/o suffix)
print( image.type )   # Type of image (e.g. jpeg)
print( image.dir )    # The directory where the ML ready data will be stored
print( image.size )   # The original size of the image
print( image.shape )  # The shape of the preprocessed image (ML ready data)
print( image.label )  # The label<jupyter_output><empty_output><jupyter_text>Let's check that we get the raw and ML ready data again.<jupyter_code>print("Raw Data", image.raw)
print("ML ready data", image.data)<jupyter_output><empty_output><jupyter_text>ThumbnailsWe can also generate a store a thumbnail of the original image with the *config* parameter thumb. In the example below, we create a thumbnail with size 16x16<jupyter_code>image = Image('gestures/1/1.jpg', 1, config=['grayscale', 'thumb=(16,16)'])
print(image.thumb)
print("Thumb Shape", image.thumb.shape)<jupyter_output><empty_output><jupyter_text>Aysnchronous PreprocessThe image data can also be preprocessed asynchronously. In this mode, the parameter *ehandler* is set to an event handler (function) that will be called when the image is done being preprocessed. The image object is passed as a parameter to the event handler.<jupyter_code>def func(image):
    print("DONE", image.name)

image = Image('gestures/1/1.jpg', 1, ehandler=func)
# Let's cleanup and remove the HDF5 file
os.remove("1.h5")<jupyter_output><empty_output><jupyter_text>Remote Image (Url)The Image class (and correspondly the Images class), paths to the image file may be specified as an URL; providing the ability to preprocess images stored at remote locations. In this case, an HTTP request is made to retrieve the image data over the network.<jupyter_code># Let's load an image from the CNN news website
image = Image('https://cdn.cnn.com/cnnnext/dam/assets/180727161452-trump-speech-economy-072718-exlarge-tease.jpg', 2)<jupyter_output><empty_output><jupyter_text>Let's look at some properties.<jupyter_code># Let's display some properties of the image that was fetched from a remote location and then preprocessed in ML ready data.
print(image.name)
print(image.size)
print(image.shape)<jupyter_output><empty_output><jupyter_text>Raw Image (Pixel)The Image class (and correspondly the Images class), paths to the image file may alternatively be the raw pixel input; providing the ability to preprocess images without retreiving from storage, when they are otherwise already in memory.<jupyter_code># import the openCV module
import cv2

# Read the pixel data into memory for an image using openCV
raw = cv2.imread('gestures/1/1.jpg')

# Let's load the image from directly the raw pixel data in memory
image = Image(raw, 1)<jupyter_output><empty_output><jupyter_text>Let's look at some properties. Note, since this is raw pixel data, the image has no name, and the size is the decompressed (raw) size in memory.<jupyter_code># Let's display some properties of the image that was directly loaded from raw pixel data and then preprocessed in ML ready data.
print(image.name)
print(image.size)
print(image.shape)<jupyter_output><empty_output><jupyter_text>Image AugmentationImage Augmentation is the process of generating (synthesizing) new images from existing images, which can then be used to augment the training process. Augmentation can include, rotation, skew, sharpending and blur of existing images. These new images are then feed into the neural network during training to augment the training set. Rotating and skew aid in recognizing images from different angles, and sharpening and blur help generalize recognition (combat overfitting), as well as recognition under different lightening and time of day conditions.The Image class supports generating new images by rotation. Any degree of rotation can be specified from 0 to 360.<jupyter_code># Let's now rotate it 90 degress
rotated = image.rotate(90)

# Let's now look at the rotated image
cv2.imshow('image',rotated)
cv2.waitKey(0)<jupyter_output><empty_output><jupyter_text>Images ClassThe Images class supports the preprocessing of a collection of images into machine learning ready data. For required parameters, the Images class takes a list of images and either a list of corresponding labels, or a single value, where all the images share the same label.Let's start by creating an Images object for all the images under the subfolder 1 (letter A).<jupyter_code># Let's get a list of all the images in the subfolder for the label 1 (letter A)
imgdir = "gestures/1/"
imglst = [imgdir + x for x in os.listdir(imgdir)]

# There should be 1200 images
len(imglst)<jupyter_output><empty_output><jupyter_text>Let's now create an Images object and preprocess all the above images.    1. Process all 1200 images in the subfolder 1    2. Set the label to 1    3. Convert them to grayscale.    By default, the image data will be stored in an HDF5 file with the name 'collection.1.h5'.<jupyter_code># Preprocess the set of images
images = Images(imglst, 1, config=['grayscale'])

# Check that the image data is stored in HDF5 file.
os.path.exists("collection.1.h5")<jupyter_output><empty_output><jupyter_text>Images PropertiesNext, we will show some properties of the Images class.Note, how fast it was to preprocess the set of 1200 images into machine ready data and store them in an HDF5 file.<jupyter_code>print( images.name )   # The name of the collection of images
print( images.dir )     # where the ML ready data is stored
print( images.time )   # The length of time to preprocess the collection of images
print( len(images) )     # The len() operator is overridden to return the number of images in the collection
# Let's print the vector of labels
print("LABELS", images.labels)<jupyter_output><empty_output><jupyter_text>The Image objects for each corresponding image can be accessed using the [] index operator. Let's get the 33rd one.<jupyter_code># The third Image object
image = images[32]
print(type(image))
print("Name", image.name)<jupyter_output><empty_output><jupyter_text>Directories (Subfolders) of ImagesThe Images class can alternately take a list of subfolders (vs. list of images); in which case, all the images under each subfolder are preprocessed into ML ready data. This is useful if your images are separated into subfolders, where each subfolder is a separate class (label) of images. This is a fairly common practice.In this case, the corresponding label in the same index of the labels parameter will be assigned to each image in the subfolder.In the example below, we also use the *name* parameter to specify a name (vs. default) for the collection.<jupyter_code># Let's process a list of subfolders of images, and name the collection 'foobar'
images = Images(['gestures/1', 'gestures/2'], [1,2], name='foobar')
# We have two subfolders of 1200 images each, so we should expect 2400 images
print(len(images))

# cleanup
os.remove('foobar.h5')<jupyter_output><empty_output><jupyter_text>Aysnchronous PreprocessA collection of images can also be preprocessed asynchronously. In this mode, the parameter *ehandler* is set to an event handler (function) that will be called when the collection of images is done being preprocessed. The Images object is passed as a parameter to the event handler.<jupyter_code>def func(images):
    print("DONE", images.name, "TIME", images.time)
    
images = Images(imglst, 1, config=['grayscale'], ehandler=func)<jupyter_output><empty_output><jupyter_text>Assemblying a CollectionFor performance purposes, one may decide to process subparts of a collection asynchronosly, and then assemble them together into a single collection. The Images class provides support for this assembling subcollections into a single collection using the overridden *+=* operator, to merge other preprocessed collections into a single collection.In the example below, we first create a collection for the letter 'A' images (label 1) and then separately create a second collection for the letter 'B' images (label 2), and use the *+=* operator to merge the second collection into the first collection.<jupyter_code># Create collection for the letter 'A' images
images = Images(['gestures/1'], 1)

# Create collection for the letter 'B' images and merge it with the letter 'A' image collection
images += Images(['gestures/2'], 2)

# The merged collection should have 2400 images (1200 for 'A', and 1200 for 'B')
print(len(images))<jupyter_output><empty_output><jupyter_text>Splitting a Collection into Training and Test DataThe *split* property will split the Image objects into training and test data. The list of training image objects is then randomized. When used as a setter, the property takes either 1 or 2 arguments. The first argument is the percentage that is test data, and the optional second argument is the seed for the random shuffle.<jupyter_code># Split the image objects into 80% training and 20% test
images.split = 0.20

# Let's verify that the training set is 80% (960 of 1200) by printing the internal variable _train
print(len(images._train))

# Let's now print the randomized list of image object indices
print("TRAIN INDICES", images._train)<jupyter_output><empty_output><jupyter_text>Let's now add the optional parameter for a random seed.<jupyter_code># Split the image objects into 80% training and 20% test
images.split = 0.20, 42

# Let's now print the randomized list of image object indices
print("TRAIN INDICES", images._train)<jupyter_output><empty_output><jupyter_text>Batch Feeding (Batch Gradient Descent)There are three ways to use the Images object to feed a neural network. In batch mode, the entire training set can be ran through the neural network as a single pass, prior to backward probagation and updating the weights using gradient descent. This is known as 'batch gradient descent'.When the *split* property is used as a getter, it returns the image data and corresponding labels for the training and test set similar to using sci-learn's train_test_split() function.<jupyter_code># Set the percentage and seed, and split the data
images.split = 0.20, 42

# Get the training, test sets and corresponding labels
x_train, x_test, y_train, y_test = images.split<jupyter_output><empty_output><jupyter_text>Let's verify and print the len of the train, test and corresponding labels.<jupyter_code>print("x_train", len(x_train))
print("y_train", len(y_train))
print("x_test", len(x_test))
print("y_test", len(y_test))<jupyter_output><empty_output><jupyter_text>Let's verify the contents that the elements are what we expect.<jupyter_code># Each element in x_train list should be a numpy array
print(type(x_train[0]))
# Each element should be in the shape 50 x 50 pixels
print(x_train[0].shape)
# Each elment in y_train should be the label (integer)
print(type(y_train[0]))<jupyter_output><empty_output><jupyter_text>Next Iterating (Stochastic Gradient Descent)Another way of feeding a neural network is to feed one image at a time and do backward probagation, using gradient descent. This is known as stochastic gradient descent.The *next()* operator supports iterating through the training list one image object at a time. Once all of the entire training set has been iterated through, it is reset and the training set is randomly re-shuffled for the next epoch.<jupyter_code># Let's iterate through the ML ready data and label for each image in the training set
while True:
    data, label = next(images)
    if data is None: break
    print(type(data), label)<jupyter_output><empty_output><jupyter_text>Mini-batch generationAnother way of feeding a neural network is through mini-batches. A mini-batch is a subset of the training set, that is greater than one. After each mini-batch is feed, then backward probagation, using gradient descent, is done. Typically, minibatches are set to sizes like 30, 50, 100, or 200. We will use the *minibatch* property as a setter to set the mini-batch size to 100.<jupyter_code># Set minibatch size to 100 images
images.minibatch = 100<jupyter_output><empty_output><jupyter_text>When we use the *minibatch* property as a getter, it will create a generator.<jupyter_code># Calculate the number of batches
nbatches = len(images) // 100

# process each mini-batch
for _ in range(nbatches):
    # Create a generator for the next minibatch
    g = images.minibatch
    # Get the data, labels for each item in the minibatch
    for data, label in g:
        pass<jupyter_output><empty_output><jupyter_text>Datset of ImagesLet's load the entire dataset - that's 27 collections of 1200 images each.<jupyter_code># Prepare each set of labeled data into machine learning ready data
# The images are 50x50, bitdepth=8, 1 channel (grayscale)
total = 0
collections=[]
for label in labels:
    # Get a list of all images in the subdirectory for this label (should be 1200 images)
    imgdir = "gestures/" + label + "/"
    imglst = [imgdir + x for x in os.listdir(imgdir)]
    images = Images(imglst, int(label), name='tmp' + label, config=['flatten', 'grayscale', 'raw'])
    collections.append(images)
    print("Procesed: " + label, "Number of images:", len(images), "Time: ", images.time)
    total += images.time
    
print("average:", total / len(labels))<jupyter_output><empty_output><jupyter_text>Let's verify the preprocessing of our image data<jupyter_code># Let's see how many batches (collections) we have (hint: should be 27)
print(len(collections))

# Let's verify that the items in the collections are an Images object
collection = collections[3]
print(type(collection))

# For a collection, let's see how many image objects we have (hint: should be 1200)
print(len(collection))<jupyter_output><empty_output><jupyter_text>Let's look at the first Image object in this collection.<jupyter_code># Let's get the first Image item and verify it is an Image object
image = collection[0]
print(type(image))<jupyter_output><empty_output><jupyter_text>Let's name view some of the properties and verify that images got processed as expected.<jupyter_code># Let's get some basic information about the image
print(image.name)  # the root name of the image
print(image.type)  # the image file suffix
print(image.size)  # the size of the image on disk
# Let's now check the raw (uncompressed) unprocessed image
print(image.raw.shape)
# Let's look at how the image got processed.
print(image.shape)  # Note, that the preprocessed image was flattened into a 1D vector. It was 50x50, and now is 2500.<jupyter_output><empty_output><jupyter_text>Let's now take a look at the image. Remember to hit any key to exit the viewer (i.e., cv2.waitKey(0))<jupyter_code># Let's view the raw image
import cv2
cv2.imshow('image',image.raw)
cv2.waitKey(0)<jupyter_output><empty_output><jupyter_text>END OF SESSION 3<jupyter_code># some cleanup
os.remove('collection.1.h5')
os.remove('untitled.h5')
os.remove('180727161452-trump-speech-economy-072718-exlarge-tease.h5')
for _ in range(27):
    os.remove('tmp' + str(_) + '.h5')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08165322580645161, 'fraction_numerical': 0.015725806451612902, 'mean_word_length': 5.2003125, 'pattern_counts': {'":': 0, '<': 157, '<?xml version=': 0, '>': 157, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '494225', 'license': 'Apache-2.0', 'n_tokens_mistral': 6030, 'n_tokens_neox': 5659, 'n_words': 2519, 'path': 'train/session 3.ipynb', 'repo_name': 'Synchronicity89/Gap', 'size': '19840'}