<jupyter_start><jupyter_text>Applying a Different Augmentation Strategy Per ClassLet's say we wished to augment the MNIST dataset, but you wished to use a generator to supply a neural network with data. Ordinarily you could write a pipeline that would augment all the data, regardless of the class. However with MNIST you might want to have different pipelines for each of the 10 different classes. For example, it would make sense to flip images for the figure 8 both horizontally and vertically and still end up with feasible data. The figure 3 could be flipped vertically but not horizontally. Conversely, the figure 4 should not be flipped either horizontally or vertically. We can do this by creating 10 different pipelines, and adding or removing the appropriate operations from each pipeline as required.Augmentor does not support this natively, but it can be performed easily enough, and here we will learn how. First we import the required libraries:<jupyter_code>import Augmentor
import numpy as np
import os
import glob
import random
import collections

from PIL import Image

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.datasets import mnist

random.seed(0)<jupyter_output>Using Theano backend.
Using gpu device 0: GeForce GTX TITAN X (CNMeM is enabled with initial size: 90.0% of memory, cuDNN not available)<jupyter_text>1. Point to a Root DirectoryYour root directory must contain subdirectories, one for each class in your machine learning classification problem:<jupyter_code>root_directory = "/home/marcus/Documents/mnist/train/*"<jupyter_output><empty_output><jupyter_text>2. Scan for folders in the root directoryWe use `glob.glob()` to scan for all files in the `root_directory` and only choose those that are directories. These will be out classes:<jupyter_code>folders = []
for f in glob.glob(root_directory):
    if os.path.isdir(f):
        folders.append(os.path.abspath(f))

print("Folders (classes) found: %s " % [os.path.split(x)[1] for x in folders])<jupyter_output>Folders (classes) found: ['6', '0', '3', '4', '7', '5', '1', '8', '2', '9']<jupyter_text>3. Create a pipeline for each classNow we create a pipeline object for each of the classes. MNIST consists of 10 digits, and each digit represents one class:<jupyter_code>pipelines = {}
for folder in folders:
    print("Folder %s:" % (folder))
    pipelines[os.path.split(folder)[1]] = (Augmentor.Pipeline(folder))
    print("\n----------------------------\n")<jupyter_output>Folder /home/marcus/Documents/mnist/train/6:
Initialised with 5918 image(s) found.
Output directory set to /home/marcus/Documents/mnist/train/6/output.
----------------------------

Folder /home/marcus/Documents/mnist/train/0:
Initialised with 5923 image(s) found.
Output directory set to /home/marcus/Documents/mnist/train/0/output.
----------------------------

Folder /home/marcus/Documents/mnist/train/3:
Initialised with 6131 image(s) found.
Output directory set to /home/marcus/Documents/mnist/train/3/output.
----------------------------

Folder /home/marcus/Documents/mnist/train/4:
Initialised with 5842 image(s) found.
Output directory set to /home/marcus/Documents/mnist/train/4/output.
----------------------------

Folder /home/marcus/Documents/mnist/train/7:
Initialised with 6265 image(s) found.
Output directory set to /home/marcus/Documents/mnist/train/7/output.
----------------------------

Folder /home/marcus/Documents/mnist/train/5:
Initialised with 5421 image(s) found.
Output [...]<jupyter_text>We can summarise what was scanned:<jupyter_code>for p in pipelines.values():
    print("Class %s has %s samples." % (p.augmentor_images[0].class_label, len(p.augmentor_images)))<jupyter_output>Class 1 has 6742 samples.
Class 0 has 5923 samples.
Class 3 has 6131 samples.
Class 2 has 5958 samples.
Class 5 has 5421 samples.
Class 4 has 5842 samples.
Class 7 has 6265 samples.
Class 6 has 5918 samples.
Class 9 has 5949 samples.
Class 8 has 5851 samples.<jupyter_text>4. Add operations to the pipelinesHere we will add operations to each of the pipelines. Some operations will be applied to all pipelines, others only to some pipelines.Here we add a rotate operation to all pipelines (and hence will be applied to all digits):<jupyter_code>for pipeline in pipelines.values():
    pipeline.rotate(probability=0.5, max_left_rotation=5, max_right_rotation=5)<jupyter_output><empty_output><jupyter_text>Here we add some operations that we only want to apply to certain classes. The figure 8 can be flipped horizontally and vertically:<jupyter_code>pipelines["8"].flip_top_bottom(probability=0.5)
pipelines["8"].flip_left_right(probability=0.5)<jupyter_output><empty_output><jupyter_text>While the figure 3 can only be flipped vertically:<jupyter_code>pipelines["3"].flip_top_bottom(probability=0.5)<jupyter_output><empty_output><jupyter_text>5. Define a class label / class integer mapThe classes will have string labels associated with them, depending on the name of each class's parent folder. Here you must map the names of each of your classes with the 0-based index (which must correspond to the test data of your dataset).In the case of MNIST this is quite easy, the samples for the digit 0 were stored in a folder 0 and have the text label 0, and so on:<jupyter_code>integer_labels = {'0': 0, 
                  '1': 1, 
                  '2': 2, 
                  '3': 3, 
                  '4': 4, 
                  '5': 5, 
                  '6': 6, 
                  '7': 7, 
                  '8': 8, 
                  '9': 9}<jupyter_output><empty_output><jupyter_text>6. Define pipeline containers to store the pipelines and additional informationLater we will need each pipeline's 0-based integer label as well as its categorical label (depending on the type of neural network you define):<jupyter_code>PipelineContainer = collections.namedtuple('PipelineContainer', 
                                           'label label_integer label_categorical pipeline generator')

pipeline_containers = []

for label, pipeline in pipelines.items():
    label_categorical = np.zeros(len(pipelines), dtype=np.int)
    label_categorical[integer_labels[label]] = 1
    pipeline_containers.append(PipelineContainer(label, 
                                                 integer_labels[label], 
                                                 label_categorical, 
                                                 pipeline, 
                                                 pipeline.keras_generator(batch_size=1)))<jupyter_output><empty_output><jupyter_text>7. Define a generator functionNeural networks in Keras can be supplied with a generator to supply training data. Because we have one generator for each pipeline, we need to create "generator of generators":<jupyter_code>def multi_generator(pipeline_containers, batch_size):
    while True:
        X = []
        y = []
        for i in range(batch_size):
            pipeline_container = random.choice(pipeline_containers)
            image, _ = next(pipeline_container.generator)
            image = image.reshape((28,28,1)) # Or (1, 28, 28) for channels_first, see Keras' docs.
            X.append(image)
            y.append(pipeline_container.label_categorical)  # Or label_integer if required by network
        X = np.asarray(X)
        y = np.asarray(y)
        yield X, y<jupyter_output><empty_output><jupyter_text>8. Create the generator objectCreate a generator, `g` to pass data randomly from each pipeline (and hence each class) to a neural network:<jupyter_code>batch_size = 128

g = multi_generator(pipeline_containers=pipeline_containers, 
                    batch_size=batch_size)  # Here the batch size can be set to any value<jupyter_output><empty_output><jupyter_text>To generate a batch of 128 images and labels, at random from a random pipeline defined above, we can use the `next()` function:<jupyter_code>X, y = next(g)<jupyter_output><empty_output><jupyter_text>We can confirm that we are receiving images in batches of 128 and that the labels correspond to the images in each pipeline:<jupyter_code>print("%s images returned. %s labels returned.") % (np.shape(X)[0], len(y))<jupyter_output>128 images returned. 128 labels returned.<jupyter_text>We can use PIL to view the augmented images and cofirm the labels match (note that PIL requires images to be specified differently to how Keras expects data, hence some preprocessing of the data must be performed):<jupyter_code>image_index = 3  # Take image index 3 from the batch

x_array = X[image_index]
x_array = x_array.reshape((28,28))
x_array = x_array * 255
x_array = x_array.astype(np.uint8)
Image.fromarray(x_array)<jupyter_output><empty_output><jupyter_text>The label below should correspond to the image output above:<jupyter_code>print("Image label: %s" % (np.nonzero(y[image_index])[0][0]))<jupyter_output>Image label: 4<jupyter_text>9. Train a neural network with the generatorLast, we train a neural network with the differing pipelines for each class.First we define a neural network:<jupyter_code>num_classes = len(pipelines)
input_shape = (28, 28, 1)

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3),
                 activation='relu',
                 input_shape=input_shape))
model.add(Conv2D(64, (3, 3), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(num_classes, activation='softmax'))<jupyter_output><empty_output><jupyter_text>Once a network has been defined, you can compile it so that the model is ready to be trained with data:<jupyter_code>model.compile(loss=keras.losses.categorical_crossentropy,
              optimizer=keras.optimizers.Adadelta(),
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Using the same batch size as the generator above, we can begin to train the neural network:<jupyter_code>h = model.fit_generator(g, steps_per_epoch=50000/batch_size, epochs=10, verbose=1)<jupyter_output>Epoch 1/10
390/390 [==============================] - 31s - loss: 0.4767 - acc: 0.8502    
Epoch 2/10
390/390 [==============================] - 31s - loss: 0.1364 - acc: 0.9595    
Epoch 3/10
390/390 [==============================] - 30s - loss: 0.1029 - acc: 0.9691    
Epoch 4/10
390/390 [==============================] - 30s - loss: 0.0879 - acc: 0.9740    
Epoch 5/10
390/390 [==============================] - 30s - loss: 0.0778 - acc: 0.9771    
Epoch 6/10
390/390 [==============================] - 30s - loss: 0.0689 - acc: 0.9784    
Epoch 7/10
390/390 [==============================] - 30s - loss: 0.0650 - acc: 0.9804    
Epoch 8/10
390/390 [==============================] - 31s - loss: 0.0632 - acc: 0.9808    
Epoch 9/10
390/390 [==============================] - 30s - loss: 0.0594 - acc: 0.9828    
Epoch 10/10
390/390 [==============================] - 30s - loss: 0.0531 - acc: 0.9845
{'fraction_non_alphanumeric': 0.1316313597682419, 'fraction_numerical': 0.04064819844287525, 'mean_word_length': 3.7189235369500215, 'pattern_counts': {'":': 1, '<': 70, '<?xml version=': 0, '>': 70, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 11, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '263641', 'license': 'MIT', 'n_tokens_mistral': 3632, 'n_tokens_neox': 3229, 'n_words': 1098, 'path': 'notebooks/Per_Class_Augmentation_Strategy.ipynb', 'repo_name': 'MustafaAlahmid/Augmentor', 'size': '11046'}