<jupyter_start><jupyter_text>Image augmentation strategies: Author: Dr. <NAME>  (CEO and Chief Imagination Officer, [Moad Computer](https://www.moad.computer)) Demo data: [Kaggle Cats Vs. Dogs Redux](https://www.kaggle.com/c/dogs-vs-cats-redux-kernels-edition)  Part 01 - [Using Keras pre-processing:](https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html) Why perform image augmentation?In order to make the most out of our few training image data, the process of "augmentation" of these images via a number of random transformations is helpful. This process feed the data to the neural network model, so that it would never see twice the exact same picture. The key advantage of implementation such an augmentation strategy is to help prevent overfitting and better generalization by the trained model.In Keras this can be done via the keras.preprocessing.image.ImageDataGenerator class. This class allows you to:* configure random transformations and normalization operations to be done on your image data during training* instantiate generators of augmented image batches (and their labels) via .flow(data, labels) or .flow_from_directory(directory). These generators can then be used with the Keras model methods that accept data generators as inputs, fit_generator, evaluate_generator and predict_generator.  Example implementation of image augmentation in Keras:<jupyter_code>try:
    import warnings
    warnings.filterwarnings('ignore')
    from keras.preprocessing.image import ImageDataGenerator
except:
    print ("Please install Keras (cmd: $sudo pip3 install keras) to run this notebook ...")
datagen = ImageDataGenerator(rotation_range=40,
                             width_shift_range=0.2,
                             height_shift_range=0.2,
                             rescale=1./255,
                             shear_range=0.2,
                             zoom_range=0.2,
                             horizontal_flip=True,
                             fill_mode='nearest')<jupyter_output><empty_output><jupyter_text>Image Augmentation in Keras -- Quick start:For more information, see the [documentation](https://keras.io/preprocessing/image/).* rotation_range is a value in degrees (0-180), a range within which to randomly rotate pictures* width_shift and height_shift are ranges (as a fraction of total width or height) within which to randomly translate pictures vertically or horizontally* rescale is a value by which we will multiply the data before any other processing. Our original images consist in RGB coefficients in the 0-255, but such values would be too high for our models to process (given a typical learning rate), so we target values between 0 and 1 instead by scaling with a 1/255. factor.* shear_range is for randomly applying [shearing transformations](https://en.wikipedia.org/wiki/Shear_mapping)* zoom_range is for randomly zooming inside pictures* horizontal_flip is for randomly flipping half of the images horizontally --relevant when there are no assumptions of horizontal assymetry (e.g. real-world pictures).* fill_mode is the strategy used for filling in newly created pixels, which can appear after a rotation or a width/height shift.<jupyter_code>from keras.preprocessing.image import ImageDataGenerator, array_to_img, img_to_array, load_img
datagen = ImageDataGenerator(rotation_range=40,
                             width_shift_range=0.2,
                             height_shift_range=0.2,
                             shear_range=0.2,
                             zoom_range=0.2,
                             horizontal_flip=True,
                             fill_mode='nearest')
img = load_img('./train/cats/cat.1.jpg')
x = img_to_array(img)
x = x.reshape((1,) + x.shape)  
i = 0
for batch in datagen.flow(x, batch_size=1,
                          save_to_dir='./preview/', save_prefix='cat', save_format='jpeg'):
    i += 1
    if i > 20:
        break<jupyter_output><empty_output><jupyter_text>Keras pre-processing overview:* The load_img uses Pillow, a complete fork of PIL. This creates a PIL image.* The img_to_array creates a Numpy array with shape (3, 150, 150).* The reshape command creates a Numpy array with shape (1, 3, 150, 150).* The .flow() command below generates batches of randomly transformed images and saves the results to the `../data/cats_dogs/preview/` directory* The break function prevents the loop from iterating indefinitely.<jupyter_code>import os
import matplotlib.image as mpl_image
import matplotlib.pyplot as plt
from IPython.display import Image as PyImage
def load_images(folder):
    images = []
    for filename in os.listdir(folder):
        img = mpl_image.imread(os.path.join(folder, filename))
        if img is not None:
            images.append(img)
    return images
def stack_plot(stack_size, folder):
    rows, cols = stack_size, stack_size
    fig,ax = plt.subplots(rows,cols,figsize=[24,24])
    i = 0
    try:
        for filename in os.listdir(folder):
            img = mpl_image.imread(os.path.join(folder, filename))
            ax[int(i/rows),int(i % rows)].imshow(img)
            ax[int(i/rows),int(i % rows)].axis('off')
            i += 1
    except:
        print ("Failed to add an image to the stacked plot ...")
    plt.show()<jupyter_output><empty_output><jupyter_text>Plotting augmented images:* Using matplotlib library.* The load_images function return a Numpy array of all the images in the folder specified in the function.* The stack_plot generates a stack of images contained inside a specific folder of size: stack_size*stack_size<jupyter_code>stack_plot(5, './preview/')<jupyter_output><empty_output><jupyter_text>Part 02 - Implementing a convolutional neural network that uses image augmentation:  Importing dependent libraries:<jupyter_code>try:
    from keras.preprocessing.image import ImageDataGenerator
    from keras.models import Sequential
    from keras.layers import Conv2D, MaxPooling2D
    from keras.layers import Activation, Dropout, Flatten, Dense
    from keras import backend as K
except:
    print ("Failed to load Keras modules. Verify if dependency requirements are satisfied ...")<jupyter_output><empty_output><jupyter_text>* Importing preprocessing.image and models functions from Keras* Importing layers function* Importing keras backend  Initialize some variables:<jupyter_code>img_width, img_height = 150, 150

train_data_dir = './train/'
validation_data_dir = './validation/'

nb_train_samples = 20000
nb_validation_samples = 5000

epochs = 50
batch_size = 16
if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)<jupyter_output><empty_output><jupyter_text>* Using img_width, img_height variables for specifying the dimensions of images to be consumed by the neural network* Initilaizing variables for location pointers to training data, validation data, train data sample size, validation data sample size, number of training epochs, number of images to be processed in each batch* Specifying a function to adjust input shape of the tensor if the image RGB data format is channels first or channels last  Build and compile a neural network: * Building a neural network model using the Sequential format in Keras* Compile the model using binary cross entropy as the loss function, RMSProp as the optimizer and accuracy as the evaluation metrics<jupyter_code>model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
              optimizer='rmsprop',
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Configuring data generators to process and feed the data to the neural network:<jupyter_code>train_datagen = ImageDataGenerator(rescale=1. / 255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)<jupyter_output><empty_output><jupyter_text>* The image augmentation configuration for training<jupyter_code>test_datagen = ImageDataGenerator(rescale=1. / 255)<jupyter_output><empty_output><jupyter_text>* Image augmentation configuration to be used for testing* This generator uses only rescaling  Creating train and validation generators:<jupyter_code>train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                    target_size=(img_width, img_height),
                                                    batch_size=batch_size,
                                                    class_mode='binary')
validation_generator = test_datagen.flow_from_directory(validation_data_dir,
                                                        target_size=(img_width, img_height),
                                                        batch_size=batch_size,
                                                        class_mode='binary')<jupyter_output><empty_output><jupyter_text>Creating a model fit generator function for training the neural network:<jupyter_code>model.fit_generator(train_generator,
                    steps_per_epoch=nb_train_samples // batch_size,
                    epochs=epochs,
                    validation_data=validation_generator,
                    validation_steps=nb_validation_samples // batch_size)<jupyter_output><empty_output><jupyter_text>Saving model weights at the end of the training session:<jupyter_code>model.save_weights('./model/first_try.h5')<jupyter_output><empty_output><jupyter_text>Part 03 - Improving classification accuracy of a neural network using transfer learning:  Importing dependent libraries:<jupyter_code>import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense
from keras import applications<jupyter_output><empty_output><jupyter_text>Defining and initializing variables:<jupyter_code>top_model_weights_path = './model/bottleneck_fc_model.h5'
train_data_dir = './train'
validation_data_dir = './validation'
bottleneck_train_path = './model/bottleneck_features_train.npy'
bottleneck_val_path = './model/bottleneck_features_validation.npy'

nb_train_samples = 2000
nb_validation_samples = 800

epochs = 50
batch_size = 16<jupyter_output><empty_output><jupyter_text>Specify the dimensions of images:<jupyter_code>img_width, img_height = 150, 150<jupyter_output><empty_output><jupyter_text>Build the VGG16 network:<jupyter_code>model = applications.VGG16(include_top=False, weights='imagenet', input_shape = (img_width, img_height,3))<jupyter_output><empty_output><jupyter_text>Define data generator:<jupyter_code>datagen = ImageDataGenerator(rescale=1. / 255)<jupyter_output><empty_output><jupyter_text>Creating a function to save bottleneck features:<jupyter_code>def save_train_bottlebeck_features(bottleneck_train_path=None):
    generator = datagen.flow_from_directory(train_data_dir,
                                            target_size=(img_width, img_height),
                                            batch_size=batch_size,
                                            class_mode="binary",
                                            shuffle=False)
    
    bottleneck_features_train = model.predict_generator(generator, nb_train_samples // batch_size)
    
    np.save(open(bottleneck_train_path),
            bottleneck_features_train)
def save_validation_bottlebeck_features(bottleneck_val_path=None):
    generator = datagen.flow_from_directory(validation_data_dir,
                                            target_size=(img_width, img_height),
                                            batch_size=batch_size,
                                            class_mode="binary",
                                            shuffle=False)
    
    bottleneck_features_validation = model.predict_generator(generator, nb_validation_samples // batch_size)
    
    np.save(open(bottleneck_val_path),
            bottleneck_features_validation)<jupyter_output><empty_output><jupyter_text>Saving bottleneck features:<jupyter_code>save_train_bottlebeck_features(bottleneck_train_path = bottleneck_train_path)
save_validation_bottlebeck_features(bottleneck_val_path = bottleneck_val_path)<jupyter_output><empty_output><jupyter_text>Creating a function to train the top model:<jupyter_code>def train_top_model(save_path=None, bottleneck_train_path = None, bottleneck_val_path = None):
    top_model_weights_path = save_path
    train_data = np.load(open(bottleneck_train_path, 'rb'))
    train_labels = np.array([0] * (nb_train_samples // 2) + [1] * (nb_train_samples // 2))

    validation_data = np.load(open(bottleneck_val_path, 'rb'))
    validation_labels = np.array([0] * (nb_validation_samples // 2) + [1] * (nb_validation_samples // 2))

    model = Sequential()
    model.add(Flatten(input_shape=train_data.shape[1:]))
    model.add(Dense(256, activation='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(1, activation='sigmoid'))

    model.compile(optimizer='rmsprop',
                  loss='binary_crossentropy', metrics=['accuracy'])

    model.fit(train_data, train_labels,
              epochs=epochs,
              batch_size=batch_size,
              validation_data=(validation_data, validation_labels))
    
    model.save_weights(top_model_weights_path)<jupyter_output><empty_output><jupyter_text>Intialize trainig session of the top model and save weights at the end of training:<jupyter_code>train_top_model(save_path=top_model_weights_path,              \
                bottleneck_train_path = bottleneck_train_path, \
                bottleneck_val_path = bottleneck_val_path)<jupyter_output><empty_output><jupyter_text>Fine tuning the model:  Load dependent libraries:<jupyter_code>from keras import applications
from keras.preprocessing.image import ImageDataGenerator
from keras import optimizers
from keras.models import Sequential
from keras.layers import Dropout, Flatten, Dense<jupyter_output><empty_output><jupyter_text>Specify dimensions of the images:<jupyter_code>img_width, img_height = 150, 150<jupyter_output><empty_output><jupyter_text>Load model weights:<jupyter_code>weights_path = './model/vgg16_weights.h5'
top_model_weights_path = './model/bottleneck_fc_model.h5'<jupyter_output><empty_output><jupyter_text>Initialize some variables:<jupyter_code>train_data_dir = './train'
validation_data_dir = './validation'

nb_train_samples = 20000
nb_validation_samples = 5000

epochs = 50
batch_size = 16

checkpointer_savepath = './model/checkpointer.h5'<jupyter_output><empty_output><jupyter_text>Build the VGG16 network:<jupyter_code>model = Sequential()
model.add(applications.VGG16(weights='imagenet', include_top=False, input_shape = (img_width, img_height,3)))
print('Model loaded ...')<jupyter_output><empty_output><jupyter_text>Build a classifier model to put on top of the V6616 convolutional model:<jupyter_code>top_model = Sequential()
top_model.add(Flatten(input_shape=model.output_shape[1:]))
top_model.add(Dense(256, activation='relu'))
top_model.add(Dropout(0.5))
top_model.add(Dense(1, activation='sigmoid'))<jupyter_output><empty_output><jupyter_text>Generate model summary:<jupyter_code>model.summary()
from keras.utils import plot_model 
import pydot 
import graphviz # apt-get install -y graphviz libgraphviz-dev && pip3 install pydot graphviz 
from IPython.display import SVG 
from keras.utils.vis_utils import model_to_dot
output_dir = './model'
plot_model(model, to_file= output_dir + '/model_top.png') 
SVG(model_to_dot(model).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>Load model weights:* It is necessary to start with a fully-trained classifier* This includes the top classifier* Initializing model weights from zero may not train the train the network successfully<jupyter_code>top_model.load_weights(top_model_weights_path)<jupyter_output><empty_output><jupyter_text>Add top model top of the Vgg16 convolutional base:<jupyter_code>model.add(top_model)<jupyter_output><empty_output><jupyter_text>Generate sumary with base VGG16 model:<jupyter_code>model.summary()
output_dir = './model'
plot_model(model, to_file= output_dir + '/model_full.png') 
SVG(model_to_dot(model).create(prog='dot', format='svg'))<jupyter_output><empty_output><jupyter_text>Freezing layers:* Freeze the first 25 layers, up to the last conv block* Weighhts become non-trainable and will not be updated<jupyter_code>for layer in model.layers[:25]:
    layer.trainable = False<jupyter_output><empty_output><jupyter_text>Compile the model:* With a SGD/momentum optimizer* Very slow learning rate.<jupyter_code>model.compile(loss='binary_crossentropy',
              optimizer=optimizers.SGD(lr=1e-4, momentum=0.9),
              metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Prepare data augmentation configuration:<jupyter_code>train_datagen = ImageDataGenerator(rescale=1. / 255,
                                   shear_range=0.2,
                                   zoom_range=0.2,
                                   horizontal_flip=True)
test_datagen = ImageDataGenerator(rescale=1. / 255)<jupyter_output><empty_output><jupyter_text>Create generator functions to handle data:<jupyter_code>train_generator = train_datagen.flow_from_directory(train_data_dir,
                                                    target_size=(img_height, img_width),
                                                    batch_size=batch_size,
                                                    class_mode='binary')
validation_generator = test_datagen.flow_from_directory(validation_data_dir,
                                                        target_size=(img_height, img_width),
                                                        batch_size=batch_size,
                                                        class_mode='binary')<jupyter_output><empty_output><jupyter_text>Implement a checkpoiting mechanism:<jupyter_code>from keras.callbacks import EarlyStopping, ModelCheckpoint

early_stopper = EarlyStopping(patience=5, verbose=1)
checkpointer = ModelCheckpoint(checkpointer_savepath,\
                               verbose=1,\
                               save_best_only=True)<jupyter_output><empty_output><jupyter_text>Load saved model:<jupyter_code>from keras.models import Model, load_model
load_from_checkpoint = True
load_from_config = False
load_model_weights = False
if load_from_checkpoint == True:
    model = load_model(checkpointer_savepath)
elif load_from_config == True:
    model = load_prediction_model(args)
    model = load_prediction_model_weights(args)
elif load_model_weights == True:
    try:
        model = load_prediction_model_weights(args)
    except:
        print ("An exception has occurred, while loading model weights ...")
else:
    model = model<jupyter_output><empty_output><jupyter_text>Train the model:<jupyter_code>model.fit_generator(train_generator,
                    steps_per_epoch=nb_train_samples // batch_size,
                    epochs=epochs,
                    validation_data=validation_generator,
                    validation_steps=nb_validation_samples // batch_size,
                    callbacks=[early_stopper, checkpointer])
model.output_shape[1:]<jupyter_output><empty_output><jupyter_text>Save the model:<jupyter_code>model.save_weights('./model/vgg16_tl.h5')<jupyter_output><empty_output><jupyter_text>Part 04 - [Using radial image transformation:](https://arxiv.org/abs/1708.04347)Deep learning models have a large number of free parameters that must be estimated by efficient training of the models on a large number of training data samples to increase their generalization performance. In real-world applications, the data available to train these networks is often limited or imbalanced. <NAME> et.al  propose a sampling method based on the radial transform in a polar coordinate system for image augmentation. This facilitates the training of deep learning models from limited source data. The pixel-wise transformation implemeted here provides representations of the original image in the polar coordinate system by generating a new image from each pixel. This technique can generate radial transformed images up to the number of pixels in the original image to increase the diversity of poorly represented image classes. Our experiments show improved generalization performance in training deep convolutional neural networks using these radial transformed images.<jupyter_code>from skimage import data
from skimage import io
import numpy as np
import math
import matplotlib.pyplot as plt
def to_gray(img):
    w, h,_ = img.shape
    ret = np.empty((w, h), dtype=np.uint8)
    retf = np.empty((w, h), dtype=np.float)
    imgf = img.astype(float)
    retf[:, :] = ((imgf[:, :, 1] + imgf[:, :, 2] + imgf[:, :, 0])/3)
    ret = retf.astype(np.uint8)
    return ret
def radial_transform(img,w,h):
    shape = im.shape

    new_im = np.zeros(shape)
    print(shape)
    print(len(shape))
    print('w',w)
    print('h',h)
    width = shape[1]
    height = shape[0]
    lens = len(shape)
    for i in range(0,width):
	    xita = 2*3.14159*i/width
	    for a in range(0,height):
		    x = (int)(math.floor(a * math.cos(xita)))
		    y = (int)(math.floor(a * math.sin(xita)))
		    new_y = (int)(h+x)
		    new_x = (int)(w+y)
		    #print(h.dtype)
		    if new_x>=0 and new_x<width:
			    if new_y>=0 and new_y<height:
				    if lens==3:
					    new_im[a,i,0] = (im[new_y,new_x,0]-127.5)/128
					    new_im[a,i,1] = (im[new_y,new_x,1]-127.5)/128
					    new_im[a,i,2] = (im[new_y,new_x,2]-127.5)/128
				    else:
					    new_im[a,i] = (im[new_y,new_x]-127.5)/128
					    new_im[a,i] = (im[new_y,new_x]-127.5)/128
					    new_im[a,i] = (im[new_y,new_x]-127.5)/128
    return new_im
im = io.imread('./preview/cat_0_1511.jpeg')
im = to_gray(im)
h = im.shape[0]
w = im.shape[1]
rt_im1 = radial_transform(im,(int)(w/2),(int)(h/2))
rt_im2 = radial_transform(im,(int)(w/4),(int)(h/4))
rt_im3 = radial_transform(im,(int)(w*0.5),(int)(h*0.75))
io.imshow(im)
io.imsave('./radial_transform/112.jpg',rt_im1)
io.imsave('./radial_transform/112.jpg',rt_im2)
io.imsave('./radial_transform/112.jpg',rt_im3)
io.show()
plt.figure(num='cats_dogs',figsize=(8,8))  

plt.subplot(2,2,1)     
plt.title('origin image')  
plt.imshow(im,plt.cm.gray)      

plt.subplot(2,2,2)    
plt.title('0.5')  
plt.imshow(rt_im1,plt.cm.gray)     
plt.axis('off')    

plt.subplot(2,2,3)    
plt.title('0.25')  
plt.imshow(rt_im2,plt.cm.gray)     
plt.axis('off')    

plt.subplot(2,2,4)    
plt.title('0.75')  
plt.imshow(rt_im3,plt.cm.gray)     
plt.axis('off')     

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09390906737986368, 'fraction_numerical': 0.0167371236304029, 'mean_word_length': 3.673991935483871, 'pattern_counts': {'":': 0, '<': 165, '<?xml version=': 0, '>': 166, 'https://': 6, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 46, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '40', 'dir': 'jupyter-structured-clean-dedup', 'id': '18389', 'license': 'MIT', 'n_tokens_mistral': 7297, 'n_tokens_neox': 6876, 'n_words': 1666, 'path': 'examples/Notebooks/01_Image_augmentation/Image_augmentation.ipynb', 'repo_name': 'rahulremanan/HIMA', 'size': '23201'}